Do math grad schools in the US create separate merit lists for each of their research groups depending on which professors are willing and capable of taking PhD students in the coming years or do they ignore the applicant's area of interest and just pick the top ones from a single list, even if certain popular research areas might be overrepresented? In the former case, I suppose applicants with less popular research interests might get an edge due to lowered competition.
Every admissions committee in the US is different and the priorities of the individuals are also different. But I'd guess that it is fairly rare that a specific area of interest has any meaningful priority.
The exception might be if a candidate expressed an uncompromising interest in an area that no one on the faculty was prepared to advise. But that would be rare at any R1 institution. Candidates are expected to be a bit flexible in their area; a bit. Also, if you are adamant that you will only work with professor X and they aren't open to taking students or likely to in a couple of years, then you have a disadvantage, though I'm not going to guess how severe.
Math isn't like lab sciences. The advisor-student relationship is usually 1-1. A student can have an advisor who is a member of a field-focused research seminar but have no contact at all with other members of that group. Also, Since most new doctoral students have only a bachelors degree, selection of a research advisor can come fairly late, after advanced coursework and passing of the comprehensive exams. (Advisors might have several students, of course.)
So, no, there are (very) unlikely to be separate merit lists. Admissions is based on a general prediction of the likelihood of success in the degree and thereafter, with other things playing a minor part. There is also little guarantee that any particular (small) sub-field of math is represented on a given admissions committee.
OTOH, if a department is known for the excellence of its research in a particular area, you do yourself no harm if that is your expressed interest.
You are mistaken that less popular areas (underrepresented in a department) could be an advantage. You need to find an interested advisor with sufficient background to advise you. People are unlikely to take on a student (other than one that is already recognized as beyond brilliant) if they have no background or interest in their topic.
Most R1 universities have a large enough and diverse enough faculty that most general areas are covered. If you are a bit flexible, you can probably find a good advisor. But it is other things that govern admissions.
1So how do math departments ensure that the admitted students will be able to find suitable advisors? Is the assumption just that students' interests are fungible enough that they will be able to make something work?– cag51 ♦Oct 9, 2022 at 22:46
1@cag51 nothing is ensured, since grad students in math in the US are often not admitted to work with a specific person from the start (there are exceptions, but you are not asking about that). A grad student who accepts an offer needs to realize that their work will be in an area of interest of some faculty, even if the student doesn't have a specific area of interest yet (except for something broad, like "geometry"). Even if a student accepting an offer wants to work with a specific person, that person might leave over the next summer (it's not frequent, but I know such cases).– KCdOct 10, 2022 at 1:40
1Sure, but if there are only 2 geometry professors and one of them has too many students already, then it might be a bad idea to admit 10 students who want to do geometry. I know there is no one-to-one matching, but I would have expected some attempt to roughly align the incoming class with the estimated hiring needs over the next few years. But I am no mathematician, I am probably missing something.– cag51 ♦Oct 10, 2022 at 4:15
2@cag51 there is some (small amount of) pressure on faculty to accommodate students, but it is not productive for anyone if a student's research is too distant from their advisor. But, in the US, I'd guess that few students start out with a problem to be solved (lack of sufficient background) and "mold" themselves to what is done in that department. In an R1, though, the department is usually large and diverse enough to cover almost all needs. It isn't a problem here, unlike, perhaps, Germany.– BuffyOct 10, 2022 at 11:28