32

For any kind of academic application (from graduate admission to professor position), recommendation letters have a major impact on the outcome. The basic idea is understandable: discovering what others think about the applicant. It can help the review committee to decide about the applicant.

However, recommendations letters cannot be statistically reliable. For instance, when all recommendation letters of an application are highly positive, this cannot guarantee that all colleagues think highly of the applicant. Instead, it can be the result of only a few friendships.

In a typical example, if someone has three socially close friends (including current colleagues, coworkers, past professors), then, his applications are always supported by strong recommendation letters. For example, one could have conflict with his entire university, but having close social connection with three persons who can recommend him.

So, why are recommendation letters relied upon so highly, given these limitations?

UPDATE: I do not mean friendly recommendation letters. I mean influence of friendship on a professional recommendation letter. As an another example, Applicant A who has good relationship with 20 professors of his department is an ideal academic with professional relationships at workplace. BUT applicant B who has serious conflicts with most of his colleagues (professors of his department), but having only three friends among his department professors will get better recommendation letters. Those three professors will write recommendation letters based on the applicant strengths by ignoring his weakness in the light of their friendship.

4
  • In many countries, students even write their own recommendation letters.
    – user774025
    Sep 22, 2013 at 8:02
  • 2
    Only letters from sources that are known to the committee are given heavy weight. Sep 23, 2013 at 9:08
  • 1
    One small thing I want to point out: no matter how unreliable recommendation letters may be (and I think overall the letter system works), there aren't many alternative measures of applicant aptitude that do better. For graduate admissions, we know GPA can easily be skewed by grade inflation, and official transcripts may fail to capture the full scope of the applicant's activities (did he do independent work for no academic credits? etc.). There are few (if any) ways to beat the flexibility of the recommendation letter in painting a complete picture of an applicant. Oct 20, 2013 at 2:33
  • The premises of the question are dubious: things don't work in such an extreme fashion, although there certainly is risk of corruption, as with anything. Dec 15, 2013 at 18:45

4 Answers 4

41

In a typical example, if someone has three socially close friends (including current colleagues, coworkers, past professors), then, his applications are always supported by strong recommendation letters.

No, that's not what "strong recommendation letter" means.

First, strong recommendation letters do not simply state the author's high personal regard for the student, but provide specific, personal, and credible detail supporting the applicant's potential for excellence. I don't just want to know that someone thinks you'll succeed—I assume they wouldn't write you a letter if they thought otherwise. I want to know why. I want compelling evidence, not mere opinion.

Second, recommendation letters have more weight if they come from credible sources. At a minimum, the author should work at a credible academic institution. The best letter writers are themselves experienced, visible, active researchers, with documented experience mentoring and/or selecting candidates for admission/hiring/promotion at departments similar to the candidate's target. For faculty promotion in my department, letters are essentially required to be from full professors, preferably in named/endowed positions, in top-10 computer science departments, and there is a strong preference for ACM/IEEE Fellows, NAE or NAS members, and major award winners (Turing, Gödel, Dijkstra, Gordon Bell, etc.).

The intersection of these two aspects of strong letters is direct comparisons with the applicant's peers. An ideal PhD recommendation letter for my department includes sentences like "Among the 13 undergraduates I have mentored who went on to top-10 PhD programs in computer science, I would rank [applicant] roughly 3rd, well below [famous person who proved P=NP], but on par with [successful person] at MIT, [successful person] at Stanford, and [successful person] at CMU."

7
  • I did not merely friends. I meant if someone had three close friends and 20 enemies from full professors of his department, he will get better recommendations than someone who had normal relationship with 23 professors of his department. Don't tell me that full professors are 100% professional in writing recommendation letters, and never consider their personal relationships.
    – Googlebot
    Sep 21, 2013 at 16:04
  • "credible academic institution": So do the NIH, DOE national labs, and NIST count as "academic institutions" by this definition?
    – aeismail
    Sep 21, 2013 at 21:25
  • 4
    @All Of course not. But lack of professionalism doesn't actually help that much; evidence is evidence. And seriously, what are the chances that someone is going to be BFFs with exactly the three people who can write in technical detail about their research, and piss off everyone else in their department? It's a small world; that sort of stuff gets out.
    – JeffE
    Sep 21, 2013 at 21:59
  • 2
    @All If by "biased" you mean "will write false things because of a social relationship," then this probably happens, but I would argue that JeffE's definition of a strong letter would be at odds with the rest of a candidate's record in those cases. If a letter-writer says great things but the rest of the record shows otherwise, then the candidate won't be particularly competitive. Sep 22, 2013 at 7:56
  • 3
    @JeffE That's one of the best descriptions of what one should look for in letters of recommendation I've ever read. The only thing I am a bit wary of is the "direct comparison". When done carefully and substantiated with clear evidence, it certainly works, but otherwise it is the hardest piece to either verify, or to make sense of. Given that Steven Krantz openly advised to write something like "Two best people are A,B. C fits in between in the sense that his research is on par with A and his teaching is on par with B" in one of his "How to..." books, there is a lot of room for "abuse" here.
    – fedja
    Sep 23, 2013 at 11:45
28

The fact that applicants choose their letter writers definitely means the letters are a biased sample of opinions. In certain cases departments take big steps to address this issue. For example, promotion and tenure committees ask for some letters from people not suggested by the candidate, precisely to avoid this source of bias. Nobody worries quite as much about graduate admissions, but there are still several safeguards:

  1. If someone writes recommendations that seem unreliable, biased, or misleading, they will lose credibility and their letters will carry less weight for everyone. This is a strong incentive to keep bias in check, since professional reputation is a critical aspect of academia, and nobody wants to be thought of as a fool or scoundrel. Of course there are still some bad recommendations, but admissions committees really keep track of credibility: when we evaluate an application, one thing we discuss is what we thought of students recommended by these letters writers in the past and what we think of their judgment.

  2. Having multiple letter writers also helps. Finding one recommender who is strikingly biased is easier than finding three.

  3. Admissions committees sometimes solicit opinions from other faculty members at the applicant's university. If the applicant's letters seem questionable or difficult to interpret, or we're concerned about credibility, then it's easy to call or e-mail to get a second opinion.

Overall, I believe the system does a good job of minimizing conscious bias or manipulation. It's not perfect, but I think you are underestimating the strength of the incentives. If an applicant can find three people with a lot of credibility who are willing to hurt their reputations by writing unreliable letters to help the applicant get admitted, then they can take advantage of the system, but this is not so easy.

If anything, I'm more worried about the safeguards being too strong. One severe drawback of the letter of recommendation system is how it handles unknown letter writers. If the committee knows nothing about a recommender, has never seen a letter from them before, and doesn't expect to in the future, then the recommender has very little credibility. If an application includes only letters from unknown recommenders, then the chances of admission may be low even if the letters all say wonderful things. This isn't fair, since some applicants just don't have access to any other recommenders, but it's unavoidable under the current system. (It's not just a matter of bias or honesty, but also of whether the recommenders are even capable of judging who would be a good candidate for admission.)

2
  • 8
    "Overall, I believe the system does a good job of minimizing bias." While I think All way overstates the case, I think that there are really systematic biases in the system of relying heavily on recommendation letters; for example, everything we know about social psychology suggest that such a system will discriminate against women, etc. just in usually subtle, imperceptible ways. Sep 23, 2013 at 17:05
  • @BenWebster: Great point! I've edited to clarify that I mean conscious bias or manipulation. Sep 23, 2013 at 18:52
15

While it may be redundant to write an answer after 3 people already have, I think the other answers have not actually tried to answer the question, but to disprove the OPs hypotheses. I agree that the OP overstates the degree of bias and favoritism in the system of using recommendation letters, but those are serious issues. As the other answers point out, a letter written largely on the basis of warm personal feelings will not be very convincing, but I think the personal attitude of the letter writer will influence the reader (does anyone doubt that it's easy to write a letter that spikes almost any candidate's chances that it is still professional and accurate?). If you read a book like "Thinking Fast and Slow", and then look at how people actually make decisions about academic hiring, you will slightly horrified at how much we ignore the plain facts of social psychology.

People rely on recommendation letters because they are monkeys (OK, actually apes, but for rhetorical purposes give me this one) whose brains evolved in one sort of social situation that encouraged certain kinds of mental heuristics. One essentially universal fact about humans is that we prefer to make individualized judgements based on intuition rather than rely on any kind of impersonal rule. We are overly confident that we know when to make an exception, and we feel better seeing a mistake made based on human misjudgment, rather than some sort of numerical calculation. This is evidenced in lots of places: for example, people actually try to actively invest in the stock market when any look at the facts shows it is essentially impossible to beat passive investing in the long run.

That said, I think for academic positions, it's also probably true that there is important information in recommendation letters which is hard to get anywhere else. In a lot of cases, I'm not sure what we would look at instead. I doubt they are going anywhere, and people will strongly resist switching to any "harder" metrics, as doing so just "feels wrong."

4

Recommendation letters for associate and full professor positions (both in hiring and promotion) do not come exclusively from writers that the candidate selects. Instead, the department will solicit letters from independent sources in the candidate's field. Furthermore, the references named by the candidate cannot have close personal or professional ties to the candidate.

4
  • 1
    that cannot be considered as recommendation letter, but external peer-review.
    – Googlebot
    Sep 23, 2013 at 11:01
  • 2
    Potayto potahto.
    – JeffE
    Sep 23, 2013 at 14:05
  • @JeffE peer-review is exactly what can be relied on, but almost all job advertisement for full professor position asks for recommendation letters by people who know the applicant, not independent review.
    – Googlebot
    Sep 23, 2013 at 18:27
  • 4
    Nobody who hires tenured faculty relies solely on letters provided by applicants. Tenure cases require external letters chosen by the target department.
    – JeffE
    Sep 24, 2013 at 1:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .