Wear whatever makes you feel comfortable within reason. Mathematicians will be judging you based on what you say and how you think.
If dressing fancy makes you uncomfortable, and less likely to perform well, dress casual [even a t-shirt and shorts in most math departments would be fine]. Similarly, if dressing down makes you feel less confident, don't do it either. Be yourself. Slacks, jeans, collared shirts [short or long sleeved], plain t-shirts, sneakers, dress shoes, dresses, skirts, blouses, shorts, suits. They are all fine.
That said, I did say within reason. It's probably best if you don't wear any clothes with offensive phrases on them. But other than that it really doesn't matter. Don't stress out about your attire.
Remember, you've been accepted. You are now trying to find an advisor. You want someone who you will get along with. If a future advisor for whatever strange reason judges you poorly based on what you like to wear, you probably don't want them as an advisor. Be yourself. You are interviewing them as much as they are interviewing you. Have fun discussing your mutual interests. Your attire is especially irrelevant in this situation [compared to the case where you haven't been admitted yet, and it's pretty irrelevant even in that situation too]
If you think what you wear won't affect how you perform, as mentioned in most of the answers, slacks and a shirt with a collar or a casual dress are pretty standard (just because they basically work in every situation from a nice restaurant to a casual walk around outside), but again try not to worry about it.