That you are good or very good at abstract mathematics such as calculus but have less innate talent for Physics, Chemistry or so on. People have talents whether through nature or nurture. It is a fact (or axiom) of life.
I am partly the opposite by the sounds of it. I was very good at GCSE and OK at A-Level mathematics (so this is college for the UK but I think up to roughly High-School in the USA system). When I got to university and started taking maths courses as part of my engineering degree, I struggled in the maths modules. All the way through my education I was good at Physics and Electronics, including at the undergraduate level, but I could not do the maths when not applied.
All this means is that I am more easily able to be an electronics engineer than a pure mathematician, and that you are possibly more easily able to be a mathematician than a scientist or engineer. You or I could try to be something else but it may be easier for us to stick with our talents and we may do better in the fields we are naturally comfortable in.
If you have generally excelled in all subjects until this point you have a new lesson to learn. All people have limits to their abilities. These may be physical or mental and things get hard when you reach the limit. Some people learn early on that math or literature or music are not easy for them, some people only learn what one of their limits is when they reach undergraduate or post-graduate study when the challenge is great enough. Almost anyone can learn almost anything but not everyone finds it so easy.
Experiencing an academic challenge that seems insurmountable for the first time can be a life altering event. You have to come to terms with your limitations or find new ways to work around them or overcome them. It is OK to find things hard. It is OK to find some things too hard. All it means is that you are not as naturally talented, or have not found the way to think about something that works for you yet, or both.
What you choose to do is up to you. What you find easier to do is not up to you. What you are interested in doing is only partially up to you. It is your life, pick what you want to try and do and go try to do it :-)
P.S. Literature was not a talent for me so I hope those with more ability can correct my poor use of language and spelling in the edits.