Wear whatever makes you feel comfortable within reason. Mathematicians will be judging you based on what you say and how you think, not what you wear. And if a future advisor for whatever strange reason judges you poorly based on what you like to wear, you probably don't want them as an advisor.
Remember, you've been accepted. You are now trying to find an advisor. You want someone who you will get along with. Be yourself. You are interviewing them as much as they are interviewing you. Have fun discussing your mutual interests. Your attire is especially irrelevant in this situation [compared to the case where you haven't been admitted yet, and it's pretty irrelevant even in that situation too for most math departments in the USA]
Slacks and a shirt with a collar or a casual dress are pretty standard just because they are versatile, and you can pretty much go to any function in them and be at ease that your outfit fits in. But really, wear whatever you would normally like to wear at school or work. Perhaps there are some exceptions, but they are rather extreme ones, like showing up in clothes with very offensive phrases on them.