Can we stop looking at recommendation letters and rely purely on "objective" measurements during the admission process?

Question

My question has different emphases from Why are recommendation letters highly relied upon?.

As we know, the letter of recommendations are heavily relied upon in the process of university/graduate school applications and job market. Sometimes it is considered the most important part of the application. Sometimes, if a referee wants to strongly recommend someone, they may even directly make a phone call to certain departments/professors.

However, as we know, the quality and importance of recommendation letters depends largely on the subjective opinions and the reputations of the referees. As a result, the system of recommendation letters sometimes encourages young scholars to spend more (and perhaps unnecessary) time using strategies to gain the favor of professional scholars (I believe there are clever ways for average students to "demonstrate" to the professors that they are strong...) or spend less time in communication to avoid bad opinions (if you can't show people that you are smart, then you'd better talk less and try not to show people that you are stupid).

Meanwhile, writing letters seems to be a heavy burden on professors. You might want to argue that this is part of their job, but shouldn't they be given more time for their research and teaching? I heard some professors write dozens of letters every year, which costs them a lot of time (excluding the time for communication with applicants).

My question is, is the letter of recommendation really an indispensable part of academic application processes? Can "objective" things like GPA (well, "transcripts" or "courses and grades" might be a more accurate measurement), test and competition scores, publications alone provide enough information to evaluate the strengths and weaknesses of applicants?

Aside: I don't know much about the application of professional positions (postdoc and tenure-track positions). However, for university/graduate school applications, can we simply raise the difficulty of standard tests (for the information of those who think SAT and GRE general and even subject tests are too easy) to better differentiate between applicants? I heard that in China, PhD applicants need to take qualifying exams BEFORE being considered.

Answer 1

The problem, in my experience in mathematics in the U.S., is that neither GPA nor GRE test/reflect what is of interest for graduate work in mathematics. So, for example, making the GRE more difficult wouldn't help anything. The point is that the modes of thought, the methodologies, the world views, that are successful for undergrads or course-based Master's students are really quite different from what is needed to do a PhD.

Of course, human beings seem to have an impulse to automate things, or make them formulaic, so that GPA and GRE have a tremendous appeal. The fact (in my observation over a few decades) that these are not good predictors of success in math grad school in the U.S. (after the first year or so) does not deter many people, who'd like things to be that simple.

That is, I think it is inconveniently the case that it is impossible to make "exams" that gauge what astute grad-program admissions committees want to know.

For postdocs and further faculty positions: to my perception, it is significantly more oppressive to have to impress journal editors and referees (who have other interests) than to impress relatively senior people in one's field. That is, the assumption that making a positive impression on senior people is a burden is missing the point. Sure, yes, it is a natural feeling, much like the reaction of teenagers to the "oppression" of their parents, but I think that senior professional academics are more competent in their roles than parents may be, statistically. No, that does not guarantee lack of bias.

As far as I understand it, in the U.S., the game is that one needs to make a good impression for a while, but, then, at a certain point, one has tenure, and, in mathematics, if one has not planned finances to depend on huge external income from grants, one can exercise one's own judgement.

The abstract-economic conflict is that there are far more people who'd like such situations than there are situations. How to decide? ...

EDIT: in light of the original questioner's comments below: the premise that grades reflect the seriousness of a student I think more reflects the assumption about what "serious" means. E.g., it is possible to very scrupulously follow instructions, and to pay careful attention to what will optimize one's grade, without caring at all about the subject of a course. And, in many math courses, the "assigned problems" (from traditional textbooks) are substantially a caricature of what the subject truly is. Often, artificial makework, to fulfill the mythical need for "lots of problems". So I am not particularly interested in performance in regard to such.

There is also the implicit premise that "mathematics is obviously objective". In a certain sense, this is relatively true, but the question really should be about good mathematics. That is, (logical) correctness is not the highest virtue, by any means. And not all details are of equal significance, by far.

So, to my mind, capacity to see a larger picture and infer wise actions from it is a very important thing to cultivate, for professional mathematicians. The basic stuff, in undergrad and beginning grad courses (in the U.S.) is so standard that there are many sources, and anyone who's interested can learn it from those sources. In particular, I tell my students to not over-invest in drilling themselves on things that in real life are easy look-ups. Rather, the general "physical" sensibility and context of questions is what one needs to reflect upon... not standard stuff. Knowing what is standard and what is not, not the power to reproduce it on command, is the desired skill.

And, so, again, performance in usual coursework or GRE and such really tells nothing about the future sensibilities of a student. Letters from astute mentors/advisors have a chance to do so.

Answer 2

You are correct that the system of letters of recommendation is pretty universally disliked, by the people who need letters, by the people who write letters, and by the people who read the letters.

But the problem is that nobody has come up with a significantly better system. In particular, let me outline a few of the things that don't work:

• We know, from experience and research, that undergraduates' GRE and GPA scores are poor indicators of success as a graduate student: There appears to be little correlation between GRE scores and the probability that a student succeeds in a graduate program. Rather, personal and interpersonal skills seem to be important determinants whether students succeed in graduate school. There do not seem to be other quantitative measures a selection committee can rely on other than personal references.

• When applying for postdoc positions and starting tenure track positions, applicants may be 1-3 and 5-15 publications, respectively. There are typically no grants or awards at this level, so publications are the only quantitative measure available. But publications are difficult to assess quantitatively: What you really want is an applicant's ability level, but what you get in a list of publications is a melange that also includes the abilities of coauthors, the question of where the papers were published (which may not always have been the best location), your inability to assess whether the best work of the applicant may still be stuck in preparation or peer review, or whether an applicant does great work but writes so poorly that their manuscripts are too often rejected (something that they may still learn, or that could be rectified with appropriate mentoring and/or coauthors). There is of course also the issue that you're going to have someone as your colleague for the next 30 years, so you want to filter out the jackasses who don't get along with folks. In other words, there are again no good quantitative measures that would assess the qualities you are looking for in applicants.

The only place where I would think that letters of recommendations are pretty pointless is for hires of people at the senior level -- say, ten or more years after their first faculty positions. By that time, there is a sufficiently long track record of publications, citations, grants, speaking invitations, etc, that one can form an objective picture of a candidate from a CV alone. These people are also typically known in their communities (and if they are not, then that's a sign as well).

For all other cases, however, I think we just don't have good "objective" measures to determine whether an applicant for anything is good or not, and so we rely on letters of recommendation.

Answer 3

A brief extra word on top of Wolfgang Bangerth's and paul garrett's excellent answers:

1) Here's a few references as a starting point for discussions of success in graduate school, and admissions:
a) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3923635/
b) https://doi.org/10.1371/journal.pone.0166742
c) https://doi.org/10.1371/journal.pone.0169121

2) The best prediction of future research success is past research success. However, especially for undergraduate research, the "objective" lines on a CV are very noisy measurements. This is partially because there are fewer publications, but more because how credit is given to students (especially undergrad researchers) is widely variable. Examples: if you are 5th author, does that mean you washed bottles or you proposed and did a side experiment that got bundled into a different paper? If you are first author of a paper, did you do the experimental design and writing, or just the data collection?

Letters are not just a pile of flattery - they can describe what the student actually did, how much insight they provided to a project, etc. This part is, in fact, basically the only detailed description of what an undergraduate has actually accomplished themselves in doing research. So it's not surprising that they are useful.

[I'm going to put one minor caveat here. There may be some signs that someone will be successful using an objective test in Mathematics. The strongest example I can think of is the Putnam exam. However, even there, there is a great deal of skepticism that it predicts research ability - since one is problem solving on the six hour scale and the other on the six month scale. See this answer: https://mathoverflow.net/questions/15848/what-to-look-for-in-applicants-to-graduate-programs-in-mathematics More importantly, though strong results are a good sign, weak results may not be a bad one - I have been told there are successful math professors who received zeros on the exam.]

Answer 4

Here are some things that reference letters provide that cannot be determined from a applicant's "statistics" (i.e. grades, number of publications, etc.) that would play an important role in their graduate admissions.

Reference letters speak to an applicant's character

In my country (Canada) it is generally expected that for a graduate degree you have informally worked out who your supervisor will be before you complete the grad school application. If you can't find a professor willing to supervise you, you generally won't be accepted.

For a professor, accepting a new student is essentially selecting a new colleague. The student is someone that the professor will have to work with for at least several years, so it is important to have some idea beforehand what kind of person the applicant is, and whether they will be a good fit for your research group. Reference letters from previous supervisors/professors/employers can tell the professor whether the person is reliable, courteous, a good team player, a good communicator, etc. Just because someone is brilliant on paper doesn't mean you would enjoy working with them.

Reference letters can provide context for an applicant's "statistics"

Sometimes, through no fault of their own, a person will have a particularly bad year. Personal tragedy, health issues, etc. can all play a role in reducing an applicant's appearance on paper by affecting the number of publications that they could produce or negatively impacting their grades. On the other hand, a person might have experienced one of the above setbacks and was still able to produce impressive statistics. A good reference letter can frame the applicant's "statistics" in a different light for the person reviewing the application.

Side-note: gain respect, not favour

In my experience it is not necessary to come up with "strategies" to "gain the favour" of respected academics. What you want is respect, not favour. The difference is very meaningful, and will be clear to anyone reading your reference letters. The favour of a narcissist (yeah, they exist in any field) can be gained by giving them compliments, but narcissists aren't good referees. You can gain a person's favour by running errands for them or giving them gifts, but again, they won't be a useful reference because they won't really know you by the qualities that make you a good applicant. Gaining respect is more challenging and it will vary from person to person. Some people respect confidence, some respect humility, some respect raw talent, etc. Everyone respects hard work (everyone worth your time anyway). Gaining respect isn't a game where you need to come up with "strategies".

Answer 5

These are actually several questions IMO:

Can we stop looking at recommendation letters [which are subjective and biased?]

Not entirely, but certainly their weight can and often is smaller than you seem to believe is universal.

Is the letter of recommendation really an indispensable part of academic application processes?

Nope. I was accepted to my current post-doc in the Netherlands with no recommendation letter. I was asked to provide references in the form of contact details though. And this way is not entirely uncommon (but I have no statistics).

My question is, Can "objective" things like GPAs, test scores, publications alone provide enough information to evaluate the strengths and weaknesses of applicants?

Grade point averages, in my opinion, are definitely not to be relied on heavily - not because of their (better) objectivity, but because they:

1. Measure something which is not as well-correlated with what you actually want to be assessing
2. Have non-subjective biases due to institute grading culture, country of origin, etc.
3. Don't cover people whose background is unusual and who might want to get into research

but for screening the non-exceptional cases, I guess some minimum GPA requirement could be used.

As for relying on publications - either you rely on numbers of publications, which is a bad measure IMHO for multiple reasons which I won't expand on here, or you have to actually look at the contents of the publications, which is probably much better but much more time-consuming.

Bottom line: IMO you can decrease the reliance on recommendation letters but you'll "pay" with an increased effect of other problems and issues.

Answer 6

In my experience this is more just a statement that the Prof. knows that student and generally thinks well of him and can recommend the students. I don't think the details matter that much, especially since those are often written by the students anyways. It's basically the same with invitation letters.

Meanwhile, writing letters seems to be a heavy burden of professors.

Is it? Many Profs seem to not have any problem with this and the others just ask for a "draft" which they usually sign. In addition it doesn't take much time to write such a letter and the students maybe worked years in the lab of the Prof. and he shouldn't spend some minutes writing a letter?

Can "objective" things like GPAS, test scores, publications alone provide enough information to evaluate the strengths and weaknesses of applicants?

They can, but they cannot tell you if the student got the support of the Profs or even know them.

For example I applied for a small grant to spend some months in a research group in another country. I talked to the Prof. there, we discussed a possible topic and cooperation and he invited me to his lab. Including a letter of invitation saying that I approached him with several ideas and he is happy to host me and I have his full recommendation shows that I did all of this and he really wants me to join his lab. This is something my publications, honors, grades,... don't show at all.

Answer 7

I wanted to add an answer to this question earlier, but as I was writing it, I realized there is no easy answer.

Research is quite complex and it's not easy to predict people's performance based on anything that could be called "objective metric". Certainly someone with perfect GRE scores and GPA=4 would seem like a great candidate for a graduate program. But, the same person could be like me and have some psychological issues that prevent them to do good research. In my case, I like working alone and I lack patience, even after so many years of training myself.

It gets even worse at later stages of career, when people evaluate you based on bibliometrics. Those sound "objective" to them, but an h-index 20 might mean you were just lucky to be working in a very strong group, while the other guy with h-index 5, who won't even get considered for a faculty interview, is close to inventing the warp drive.

Letters of recommendations seem to me a good way to offset some of what those objective metrics tell about people. If you are the reader of such letters, you should look for facts confirming or not your evaluation of the candidate's ability based on what you know from their test scores, published results, etc. Those letters also tell the interviewer how the candidate works with people. They are not sufficient and should be doubled by an interview.

Even with the interview, you still cannot fully predict how the candidate will fare in the future. You might get a stellar candidate, but for whatever reason they end up working with someone they can't get along with and end up wasting years in graduate school or as postdocs without any serious result. You could also get a regular candidate who ends up working with a group in which they fit well and their career takes off. I have seen plenty of my "average" colleagues becoming great researchers simply because they were wise or lucky to choose the right people to work with. I'm not saying they didn't put in the effort, I'm saying that others with better grades and test scores didn't fare as well simply because they weren't in the right environment.

There is another reason why people need letters of recommendations. There are people who simply don't fit in the system, but would do well as researchers. I knew a student who was building robots. He didn't have any guidance, he learned everything from books and forums. His GPA was slightly above what was needed to pass, SAT was abysmal, so no normal undergraduate program would consider him. But he had a great recommendation letter from his science teacher who knew about his robots, so he got into an undergraduate program. Now he's doing well as a graduate student, and has quite a research record.

Answer 8

The problem becomes clearer when you compare what someone sifting students will probably be looking for.

At postgraduate level, you want to know what their prospects are at research. But many graduates haven't done formal research as such, and is really hard to design a grade system to rate their prospects as a future researcher.

They will also have to move much more to self development and be able to collaborate with others _- the grade system might rate how much they know, but not what kind of person they are like to work with.

Postgrads often do some level of teaching in the field. In an office they might mentor or supervise others (in some cases). Again, grades don't really tell the recruiter how they are likely to be as a person (not just in terms of technical knowledge) at this.

For similar reasons a whole bunch of other skills that are fundamental to deciding who to accept, just cannot be assessed on grading, and even assessing by interview performance is often missing key information.

The same goes for commercial as well as research roles - businesses will want to reduce risks of a problem, and the information let's them improve their chances. So a bias favouring candidates with such information will often exist.

The human perspective, by someone else, independent, who knows the person, is still extremely valuable and key to many decisions of this kind.

Answer 9

It seems most of the answers talk in support of letter of recommendations. I'll put in my views which are against it. It also seems to me that most people do not have alternatives to letter of recommendations, so I'm planning to present some idea I had through my experience with grad school applications.

Let me make one thing clear, I don't agree with the idea that GRE or GPA has good correlation with grad school success. Some studies which are in alignment with my argument are given below:

1. The Limitations of the GRE in Predicting Success in Biomedical Graduate School - Moneta-Koehler et al.
2. Multi-institutional study of GRE scores as predictors of STEM PhD degree completion: GRE gets a low mark - Petersen et al.

Of course there are more. I'll also provide one study that has a different take on this:

1. Can the GRE predict valued outcomes? Dropout and writing skill - Bridgeman et al.

I may add more studies if asked but these will suffice for talking about GRE goes into a tangent.

The primary issue with letter of recommendation is with the recommender and their biases. The country I'm from, finding recommender who's willing to give you a decent letter of recommendation is more difficult that clearing a competitive/standardized test. For example a lot of the senior faculty have an innate picture of what they perceived to be a worthy candidate. And whether you like it or not human biases cannot be removed. On top of that for a student from a rural college it is extremely difficult to network with a professor and impress them enough to get a recommendation letter out of them. A lot of the professors don't reply to emails, a lot of them don't understand the perspectives and struggles of individual students. You can sweep it under the rug saying that they are busy or that one has to be pragmatic about these issues, but then at the end of the day the one who's going to suffer is the student. There are other issues like the affiliation of the recommender. A more prestigious and elite affiliation can have huge impacts on the admission results or at least the notion of how the candidate's worth is seen. As an example from my personal experience, a friend of mine recieved his recommendations from foreign faculties even though they barely knew him and his work. It fetched him admission in some of the top grad programs in Europe. Another friend who had a much better research portfolio was struggling to get admission in Europe and his recommenders were more grounded. I had worked with both of the candidates and I know their potential. I also agree there are many more factors at play here but you have to acknowledge the blatant abuse of the system. Is that not wrong? On top of that it is often the case that when the recommender knows that the student is going to see the letter that they are more lenient while if the student cannot see the letter, they will inevitably be more critical. Some of my arguments are in alignment with the studies:

1. The Use of Reference Reports in Personnel Selection: A Review and Evaluation - Muchinsky.
2. Predicting Performance with Letters of Recommendation - Aamodt et al.

In fact there are plenty of issues discussed in these that I haven't even touched upon. The second study here has suggests improvements over the traditional letter of recommendation. My suggestion however is to completely get rid of the recommendation system. Instead I propose the following:

1. Public Peer Review: Grad school applicants should have a public profile that can be monitored or checked. As an example consider stackexchange. A student who's applying to work on, say, quantum field theory should have a stackexchange portfolio where they could answer non-textbook questions regarding the subject in sufficient detail. It is similar to how some industrial applications look for ranks in coding competitions.
2. Writing Assignments: Instead of simply relying of GRE the candidate can be given a few months to submit a graduate level essay/research article on a basket of topics pertaining to the subject of interest.
3. CV and SOP: Where their research experience and role should be emphasized and possible publications should be mentioned.
4. Interview: Where the candidate should be asked question from their writing assignment and from their research experience. The interview can be as grilling as possible to ascertain that the candidate actually knows what they have written.

As a plagiarism/outsourcing check, extremely non trivial assignment topics have to be given, where only publication (or at least high) quality essays/articles should be entertained.

Another possible alternative is to reduce the number of recommendation from 3 to 2 or 1 and then putting less emphasis on the recommendation letter and more on the interview.

Answer 10

The PageRank algorithm ranks a page as more authoritative if many pages link to it, and a page as more relevant if an authoritative page links to it.

On Stackexchange, users earn a reputation from the quality of their answers, as conferred by fellow users, which lends weight to their subsequent answers.

In academia, professors earn respect by the quality of their work, and the testimony of a respected expert in the field transfers respect onto whoever they recommend.

I think it's just human nature to be swayed by anecdotal arguments from authority more than statistical ones. To wit, an anecdote: I've submitted recommendations to a system where I had to check boxes for how unique an individual was in several criteria. The levels of uniqueness are based on how frequently I have encountered students at their level ("top 10% each year," or "top 2 or 3 in my career", etc). The criteria are things like "oral communication," "independent thinking," "personal responsibility", "leadership", etc. This could result in some statistical measure of the quality of an applicant.

I hate these matrices! I might have an opinion on a few of the criteria, but on others I really couldn't say. And I've taught maybe 5000 students in my career; asking me to arrange them in increasing order of ability makes me tired just thinking about it. So I'd much rather upload a letter, in which I give quantitative data on the student's performance, and personal qualities which show aptitude for whatever they're applying to.