Provided you are cordial and tread cautiously, no, it won't. (I know this is a late answer, but since people think of doing this fairly often, I figured I might give my two cents. While I am not a professor, I figured it might help to have a current graduate student's perspective.)
I myself have transferred between math graduate programs at different schools. Furthermore, I know three or four others who have transferred between math programs at different schools, two people who have transferred from a math program to a non-math program at a different school, and two people who have left math programs for a non-math program at the same university. (And no, these people weren't all transferring to or from the same university.) I know many others who have changed advisors or left graduate school entirely. And people often transfer for reasons other than the reputation of the schools, myself included. (Such reasons may include moving to be closer to significant others, dissatisfaction with the geographical location of the school, changing research interests, advisors switching schools and bringing their students with them, advisors ceasing to work with students, and interest in a specific advisor at a different school, among others.) As much as it would be ideal for all graduate students to arrive precisely where they belong and know exactly what they will do over the next four years, it's probably an understatement to say that this doesn't always happen. And many professors are sympathetic and understanding.
I would, however, caution you about the following. Keep in mind that if you don't get accepted into any transfer schools, you probably want to still get good funding and work with an advisor that fits you at your current school. You may indeed later find that you want to stay at your current school, even if you've been accepted to other places! So, informing those who determine your funding that you're thinking of transferring before you actually have been accepted into another school doesn't necessarily seem like a good idea to me. Depending on the perspective of the department, it can be like telling your romantic partner that you're thinking of breaking up well before you make the actual decision. (On the other hand, they might appreciate being informed early and may be very kind and understanding. It's tough to judge, since it all depends on the people involved.)
But if you do get accepted into a transfer institution and decide to go there, you should make sure to have a conversation with the heads of your current graduate program and department, to notify them that you are leaving, and to politely express why you are making this change. I'd recommend being brief (you don't need to tell them your life story), but clear, and always polite. It may be a good idea not to hold off the decision until the last minute (i.e. the April 15th deadline) like I did. Keep in mind that they need to find someone to replace you, and the closer to the deadline, the harder it is for the school to find qualified candidates.
And I agree that you should be sure not to denigrate your current institution in any of your applications, or when you tell people at your current institution why you're leaving. I also agree that when you apply to other institutions, you should very honestly express why you're making this switch (without denigration, of course).
Edit: Also, regarding Anonymous Mathematician's comment under the question: I transferred after my second year and it's worked out fine, but certainly it is best to be actively pursuing a backup plan for research at your current institution if you're attempting to transfer any later than your first year. That is, keep looking for possible advisors and research projects at your current university even after you've submitted your transfer applications.