Why are recommendation letters highly relied upon?

Question

For any kind of academic application (from graduate admission to professor position), recommendation letters have a major impact on the outcome. The basic idea is understandable: discovering what others think about the applicant. It can help the review committee to decide about the applicant.

However, recommendations letters cannot be statistically reliable. For instance, when all recommendation letters of an application are highly positive, this cannot guarantee that all colleagues think highly of the applicant. Instead, it can be the result of only a few friendships.

In a typical example, if someone has three socially close friends (including current colleagues, coworkers, past professors), then, his applications are always supported by strong recommendation letters. For example, one could have conflict with his entire university, but having close social connection with three persons who can recommend him.

So, why are recommendation letters relied upon so highly, given these limitations?

UPDATE: I do not mean friendly recommendation letters. I mean influence of friendship on a professional recommendation letter. As an another example, Applicant A who has good relationship with 20 professors of his department is an ideal academic with professional relationships at workplace. BUT applicant B who has serious conflicts with most of his colleagues (professors of his department), but having only three friends among his department professors will get better recommendation letters. Those three professors will write recommendation letters based on the applicant strengths by ignoring his weakness in the light of their friendship.

Answer 1

In a typical example, if someone has three socially close friends (including current colleagues, coworkers, past professors), then, his applications are always supported by strong recommendation letters.

No, that's not what "strong recommendation letter" means.

First, strong recommendation letters do not simply state the author's high personal regard for the student, but provide specific, personal, and credible detail supporting the applicant's potential for excellence. I don't just want to know that someone thinks you'll succeed—I assume they wouldn't write you a letter if they thought otherwise. I want to know why. I want compelling evidence, not mere opinion.

Second, recommendation letters have more weight if they come from credible sources. At a minimum, the author should work at a credible academic institution. The best letter writers are themselves experienced, visible, active researchers, with documented experience mentoring and/or selecting candidates for admission/hiring/promotion at departments similar to the candidate's target. For faculty promotion in my department, letters are essentially required to be from full professors, preferably in named/endowed positions, in top-10 computer science departments, and there is a strong preference for ACM/IEEE Fellows, NAE or NAS members, and major award winners (Turing, Gödel, Dijkstra, Gordon Bell, etc.).

The intersection of these two aspects of strong letters is direct comparisons with the applicant's peers. An ideal PhD recommendation letter for my department includes sentences like "Among the 13 undergraduates I have mentored who went on to top-10 PhD programs in computer science, I would rank [applicant] roughly 3rd, well below [famous person who proved P=NP], but on par with [successful person] at MIT, [successful person] at Stanford, and [successful person] at CMU."

Answer 2

The fact that applicants choose their letter writers definitely means the letters are a biased sample of opinions. In certain cases departments take big steps to address this issue. For example, promotion and tenure committees ask for some letters from people not suggested by the candidate, precisely to avoid this source of bias. Nobody worries quite as much about graduate admissions, but there are still several safeguards:

1. If someone writes recommendations that seem unreliable, biased, or misleading, they will lose credibility and their letters will carry less weight for everyone. This is a strong incentive to keep bias in check, since professional reputation is a critical aspect of academia, and nobody wants to be thought of as a fool or scoundrel. Of course there are still some bad recommendations, but admissions committees really keep track of credibility: when we evaluate an application, one thing we discuss is what we thought of students recommended by these letters writers in the past and what we think of their judgment.

2. Having multiple letter writers also helps. Finding one recommender who is strikingly biased is easier than finding three.

3. Admissions committees sometimes solicit opinions from other faculty members at the applicant's university. If the applicant's letters seem questionable or difficult to interpret, or we're concerned about credibility, then it's easy to call or e-mail to get a second opinion.

Overall, I believe the system does a good job of minimizing conscious bias or manipulation. It's not perfect, but I think you are underestimating the strength of the incentives. If an applicant can find three people with a lot of credibility who are willing to hurt their reputations by writing unreliable letters to help the applicant get admitted, then they can take advantage of the system, but this is not so easy.

If anything, I'm more worried about the safeguards being too strong. One severe drawback of the letter of recommendation system is how it handles unknown letter writers. If the committee knows nothing about a recommender, has never seen a letter from them before, and doesn't expect to in the future, then the recommender has very little credibility. If an application includes only letters from unknown recommenders, then the chances of admission may be low even if the letters all say wonderful things. This isn't fair, since some applicants just don't have access to any other recommenders, but it's unavoidable under the current system. (It's not just a matter of bias or honesty, but also of whether the recommenders are even capable of judging who would be a good candidate for admission.)

Answer 3

While it may be redundant to write an answer after 3 people already have, I think the other answers have not actually tried to answer the question, but to disprove the OPs hypotheses. I agree that the OP overstates the degree of bias and favoritism in the system of using recommendation letters, but those are serious issues. As the other answers point out, a letter written largely on the basis of warm personal feelings will not be very convincing, but I think the personal attitude of the letter writer will influence the reader (does anyone doubt that it's easy to write a letter that spikes almost any candidate's chances that it is still professional and accurate?). If you read a book like "Thinking Fast and Slow", and then look at how people actually make decisions about academic hiring, you will slightly horrified at how much we ignore the plain facts of social psychology.

People rely on recommendation letters because they are monkeys (OK, actually apes, but for rhetorical purposes give me this one) whose brains evolved in one sort of social situation that encouraged certain kinds of mental heuristics. One essentially universal fact about humans is that we prefer to make individualized judgements based on intuition rather than rely on any kind of impersonal rule. We are overly confident that we know when to make an exception, and we feel better seeing a mistake made based on human misjudgment, rather than some sort of numerical calculation. This is evidenced in lots of places: for example, people actually try to actively invest in the stock market when any look at the facts shows it is essentially impossible to beat passive investing in the long run.

That said, I think for academic positions, it's also probably true that there is important information in recommendation letters which is hard to get anywhere else. In a lot of cases, I'm not sure what we would look at instead. I doubt they are going anywhere, and people will strongly resist switching to any "harder" metrics, as doing so just "feels wrong."

Answer 4

Recommendation letters for associate and full professor positions (both in hiring and promotion) do not come exclusively from writers that the candidate selects. Instead, the department will solicit letters from independent sources in the candidate's field. Furthermore, the references named by the candidate cannot have close personal or professional ties to the candidate.