Top math grad programs in the U.S. will receive tons of applications from people with excellent grades in grad or advanced undergrad math classes, high math subject GRE scores, and some research. No doubt there are other factors like competition results or awards from the student's department, but they tend to carry little weight, being considered as not very predictive of performance in math research. Not to say that doing well in a competition is useless, but the marginal predictive utility of doing well vs doing extremely well is insignificant in the sense that you're not gaining much info, if any at all, about future performance when comparing top 50 in Putnam and IMC to top 10. It will just be seen as the difference between solving constructed problems well and solving constructed problems very well by putting in even more effort practicing them.
Judging the research has its own problems with verifying how much the student learned and did on their own, so admissions committees turn to recommendation letters to distinguish between very good students who have nearly maxed out all the other criteria. However, recommendation letters are qualitative in nature and can only be justified on the basis that they provide info on a candidate's ability to succeed that other factors don't. In other words, recommendation letters aren't perfect but it would be much harder to make decisions without them just using other factors, which I agree with.
All that being said, why don't U.S. programs use entrance exams as a factor on par with classes, research, and letters? They shouldn't have the bulk of the weight as that may lead to similar situations as in India and China, where students go all out just for the exam at the expense of other skills, but there should at least be some weight. Something like the preliminary exams that colleges have for 1st and 2nd year students in linear algebra, abstract algebra, real analysis, differential equations etc. could be given to applicants instead of waiting for the students to enter and then requiring them to pass those exams, which is done at almost every university.
Another idea is interviews or oral exams, but the biggest reason why those may not be used is the amount of time required to administer interviews to 25-50 applicants who have passed all the checks on previous factors (good enough classes, grades, and letters) and compare those interviews. In comparison, the effort needed to write, proctor, and grade exams can be spread out. Staff outside the committee can proctor and grade while the committee just has to come up with the questions and look at the results.