It might be easier to advance well in a more scintillating, rigorous department. But there's a lot to be said for being a big fish in a little pond, for example you might find more personal attention in the little pond. And it's easier to stand out (make a big splash!).
If you want to transfer, in principle, after the sophomore isn't a bad time to do it. But check deadlines carefully.
Will you run out of courses to take at your current college? In some locations, one can take a couple of courses from a nearby more rigorous department or a larger department in the senior year. You might not be able to transfer those credits in; but that might not cause problems if you've collected enough credits at your home institution (without adding in the transfer credits) in order to graduate. If you know that the transfer credits won't be accepted, you might want to save some money by auditing when you register for the summer class(s).
It might be worthwhile to check if your college has a foreign exchange program.
Let's say you finish out the degree where you are, but don't get into the level of program you would like in the first round. In this case, it can be helpful to take some additional classes somewhere with a strong teaching record, as a non-matriculated student, to build up your knowledge and experience.
I can't swear to this in the context of a pure math programs, but in my experience with applying to a top-notch computer science department, the undergraduate institution's name didn't matter. They were looking at the transcript, essay and recommendations. I think for math, the GRE would be on that list as well. If you can get some undergraduate research experience (possibly this coming summer).
Would the delay (compared to your peers) hold you back? Not necessarily. Many people think the undergraduate degree is some kind of race. It isn't.