Wear whatever makes you feel comfortable within reason. Mathematicians will be judging you based on what you say and how you think.
If dressing fancy makes you uncomfortable, and less likely to perform well, dress casual [even a t-shirt and shorts in most math departments would be fine]. Similarly, if dressing down makes you feel less confident, don't do it either. Be yourself. Slacks, jeans, collared shirts [short or long sleeved], plain t-shirts, sneakers, dress shoes, dresses, skirts, blouses. They are all fine.
That said, I did say within reason. It's probably best if you don't wear a tank top or middrift, or any clothes with offensive/controversial phrases on them. But other than that it really doesn't matter. Don't stress out about your attire, focus on preparing the content part of your interview.
If you think what you wear won't affect how you perform, as mentioned in most of the answers, slacks and a shirt with a collar or a casual dress are pretty standard (just because they basically work in every situation from a nice restaurant to a casual walk around outside), but again try not to worry about it.