Do math grad schools in the US create separate merit lists for each of their research groups depending on which professors are willing and capable of taking PhD students in the coming years or do they ignore the applicant's area of interest and just pick the top ones from a single list, even if certain popular research areas might be overrepresented? In the former case, I suppose applicants with less popular research interests might get an edge due to lowered competition.
Every admissions committee in the US is different and the priorities of the individuals are also different. But I'd guess that it is fairly rare that a specific area of interest has any meaningful priority.
The exception might be if a candidate expressed an uncompromising interest in an area that no one on the faculty was prepared to advise. But that would be rare at any R1 institution. Candidates are expected to be a bit flexible in their area; a bit. Also, if you are adamant that you will only work with professor X and they aren't open to taking students or likely to in a couple of years, then you have a disadvantage, though I'm not going to guess how severe.
Math isn't like lab sciences. The advisor-student relationship is usually 1-1. A student can have an advisor who is a member of a field-focused research seminar but have no contact at all with other members of that group. Also, Since most new doctoral students have only a bachelors degree, selection of a research advisor can come fairly late, after advanced coursework and passing of the comprehensive exams. (Advisors might have several students, of course.)
So, no, there are (very) unlikely to be separate merit lists. Admissions is based on a general prediction of the likelihood of success in the degree and thereafter, with other things playing a minor part. There is also little guarantee that any particular (small) sub-field of math is represented on a given admissions committee.
OTOH, if a department is known for the excellence of its research in a particular area, you do yourself no harm if that is your expressed interest.
You are mistaken that less popular areas (underrepresented in a department) could be an advantage. You need to find an interested advisor with sufficient background to advise you. People are unlikely to take on a student (other than one that is already recognized as beyond brilliant) if they have no background or interest in their topic.
Most R1 universities have a large enough and diverse enough faculty that most general areas are covered. If you are a bit flexible, you can probably find a good advisor. But it is other things that govern admissions.