Mathematics is a difficult subject that requires a lot of work. For me, this was a large part of the attraction. I want to do something challenging.

You mention physics. My first degree is in physics. I was amazed by how many people dropped out in the first year as the "wanted to study physics and not mathematics". The simple truth is that physics requires a lot of mathematics. Often this mathematics may not be very rigours nor will you often need to get too involved in deep proofs. (Unless you take a more mathematical view point). Mathematics and physics are greatly intertwined.

Don't feel stupid. For most of us mortals it is difficult and takes a lot of time.