I got the following response from a mathematics professor at UT Austin:
Gaining admission is more about the positive things in your file than about checking all the right boxes. If you haven't taken all of the standard courses, but have learned it all on your own, that's OK. However, there has to be a very good reason why you're better than another applicant who HAS taken all of the standard courses. Your letter-writers have to make a convincing argument that there is something very, very special about you. If your physics professors are better able to present these arguments than you math professors, then it's OK to have most of your letters come from physicists. But those letters have to be very, very good.
My situation is such that since I am not a mathematics major, I haven't done a lot of standard math courses, which I intend to/or have studied on my own. Clearly, I need to win the graduate admissions committee's over by making sure my letter writers write good letters for me.
My question is:
Generally, what can students like myself do to make sure they get the best possible letters from their professors? That is, how should one structure the self study plan to ensure maximum possible output and in return, the best possible letter from an instructor. More so, I'm not sure what qualifies as a "very, very special letter." What makes a letter "special?" More so, how can students like myself make sure that they are able to get such letters.