Why don't things work with comparisons in reference letters and refereeing?

Question

This recent question How high should I rate a good student for PhD admission? is just an example of how puzzling the problem of writing reference letters is.

One knows that there is inflation in the field, so they feel they have to adapt, but it is not clear how much inflation is appropriate and when it is too much. Negative comments are often omitted, and this gives the evaluators less data.

There is an analogous problem when refereeing papers, or serving as anonymous referees for research proposals and grants. This seems to be a flaw which is implicit in the evaluation method. Apart from this voluntary inflation bias, there is also an involuntary component: if I ask you to evaluate your research field/colleague/student is, you will unconsciously feel biased towards it/him/her.

Is this perceived as a big problem? If so, why is there no attempt to obtain more reliable results? Some possible strategies, although difficult to apply, spring to my mind:

• if there are enough data points for the recommender, normalize everything and "grade on a curve".

• don't ask "how good X is?"; give them two paper/applicants/projects to evaluate, and ask "which one is better, X or Y?". This makes it more difficult to find good referees, because they have to be familiar with two of the persons/projects under evaluations.

• force the evaluators to provide at least 3 positive and 3 negative points, the most outstanding ones. This won't affect the inflation bias, but at least it provides some more data on the negative aspects.

I have seriously considered writing a standard addendum to enclose to my letters and reports, stating that I realize how biased and unreliable these things are, I am in a difficult position giving an absolute score on a scale that is totally unknown to me and the other evaluators, and I'd really feel more comfortable with a comparison-based evaluation system. Would this be appropriate, or raise some eyebrows on the trustworthyness of my evaluations?

Answer 1

I suggest that an academic committee, sitting in consideration of an applicant to their PhD programme, would very likely view your addendum as being an honest attempt to preserve your set of professional ethics with regard to fairly ranking the applicant.

I do not think that the academic committee would doubt your trustworthiness. However, I expect that the committee would view whatever ranking you provided on the Good/Great/Excellent/Outstanding/Genius scale in light of your action of submitting your addendum.

Therefore the committee are not going to doubt your ability to rank the applicant, but are going to doubt the veracity of your ranking on the artificial scale.

However, I would hope that any sensible academic committee is going to be aware of the problem that we are discussing here and are therefore going to take a scaled single-valued ranking of any applicant with a massive grain of salt. They must know that a single value rank is going to be useless when judging the suitability of an applicant for a PhD programme.

Thus I would hope that, with or without your addendum, a sensible academic committee would take very little notice of the single value ranking and pay much more attention to the substantive letters of recommendation.

The addition of your addendum would not, I think, damage the applicant's application before a reasonable academic committee, as it wouldn't be news to the committee that such single-valued rankings are untrustworthy. To a reasonable academic committee, your actions of submitting the addendum might prompt others to do the same, and the practice of demanding a single-value metric abandoned over time. To an academic committee that places great store in such single-value metrics, your addendum sets out your concerns and presumably instructs the committee how you the single-value ranking is to be interpreted.

Answer 2

I wouldn't worry too much about it. Good schools will rely primarily on the content of your letter. Focus on writing a helpful letter. Include comparisons to other students who you have worked with in the past, where possible. That will enable people to calibrate your letter.

I wouldn't don't worry too much about the numeric scores. Good schools won't pay too much attention to the numeric scores, if you've done a good job with the body of the letter and written a detailed, helpful letter with plenty of concrete specifics. Any school that puts more weight on the numeric score than the body of the letter is just screwing themselves by using bad decision-making processes. You can't do anything about that; you can't fix other peoples' bad judgement. All you can do is do the right thing when it is under your control, and leave it at that.