20

My question has different emphases from Why are recommendation letters highly relied upon?.

As we know, the letter of recommendations are heavily relied in the process of university/graduate school applications and job market. Sometimes it is considered the most important part of the application. Sometimes if a referee wants to strongly recommend someone, s/he may even directly makes a phone call to certain departments/professors.

However, as we know, the quality and importance of recommendation letters depends largely on the subjective opinions and the reputations of the referees. As a result, the system of recommendation letters sometimes encourages young scholars to spend more (and perhaps unnecessary) time using strategies to gain the favor of professional scholars (I believe there are clever ways for average students to "demonstrate" to the professors that they are strong...) or spend less time in communication to avoid bad opinions(if you can't show people that you are smart, then you'd better talk less and try not to show people that you are stupid).

Meanwhile, writing letters seems to be a heavy burden of professors. You might want to argue that this is part of their job, but shouldn't they be given more time on their research and teaching? I heard some professors write dozens of letters every year which costs them a lot of time (excluding the time for communication with applicants).

My question is, is the letter of recommendation really an indispensable part of academic application processes? Can "objective" things like GPA(well, "transcripts" or "courses and grades" might be a more accurate measurement), test and competition scores, publications alone provide enough information to evaluate the strengths and weaknesses of applicants?

Aside: I don't know much about the application of professional positions (postdoc and tenure-track positions). However, for university/graduate school applications, can we simply raise the difficulty of standard tests(for the information of those who think SAT and GRE general and even subject tests are too easy) to set the applicants apart from each other better? I heard in China, Ph.D. applicants need to take qualifying exams BEFORE being considered.

  • 3
    @BlaB You are right, but I think compared with industry, academia need more people who will focus on teaching and research instead of social networking whose main purpose (for some people) is to gain the extra "advantages" which don't have much to do with one's true academic capabilities. – No One Oct 3 '17 at 19:58
  • 2
    I am sorry to disagree. I think academia need more people who are able to communicate more efficiently, network and create links with the industry. People in academia, sometimes especially in the hard sciences, tend to neglect soft skills a bit too much. The industrial reality (at least in science), requires a lot of soft skills. If you acquire those as a professor, you can more efficiently transmit them to your students. – BlaB Oct 3 '17 at 20:00
  • 40
    After you have worked with, much less managed, some reasonable number of people, you will (or at least should) discover that human capacity and human performance can never be captured with your 'objective' measurements. There are other factors, including soft skills, that directly impact how well people perform. Personal recommendations will always be sought after, and recommendations from people you trust are worth their weight in gold out here in the real world. – Jon Custer Oct 3 '17 at 20:54
  • 43
    For grad schools in particular, in some fields very few students will have publications, so that's out. And at top schools, you may have a huge number of applicants who all have GPAs between 3.98 and 4.0, and GRE scores above the 95th percentile. You cannot distinguish statistically between such applicants. So mathematically, no, these objective measures are not sufficient. – Nate Eldredge Oct 3 '17 at 22:02
  • 5
    As someone who was lucky enough to make it into a top-5 school in a relatively competitive field, I had a few talks with the admission committee afterwards and they explicitly mentioned that they trusted the letters much more than scores, as the comparability between universities was extremely hard. I was personally only asked about my one bad grade from my masters - as others mentioned, everyone who applies are pretty great, it is the other factor that matter it seems. – Forgottenscience Oct 3 '17 at 22:27

10 Answers 10

41

The problem, in my experience in mathematics in the U.S., is that neither GPA nor GRE test/reflect what is of interest for graduate work in mathematics. So, for example, making the GRE more difficult wouldn't help anything. The point is that the modes of thought, the methodologies, the world views, that are successful for undergrads or course-based Master's students are really quite different from what is needed to do a PhD.

Of course, human beings seem to have an impulse to automate things, or make them formulaic, so that GPA and GRE have a tremendous appeal. The fact (in my observation over a few decades) that these are not good predictors of success in math grad school in the U.S. (after the first year or so) does not deter many people, who'd like things to be that simple.

That is, I think it is inconveniently the case that it is impossible to make "exams" that gauge what astute grad-program admissions committees want to know.

For postdocs and further faculty positions: to my perception, it is significantly more oppressive to have to impress journal editors and referees (who have other interests) than to impress relatively senior people in one's field. That is, the assumption that making a positive impression on senior people is a burden is missing the point. Sure, yes, it is a natural feeling, much like the reaction of teenagers to the "oppression" of their parents, but I think that senior professional academics are more competent in their roles than parents may be, statistically. No, that does not guarantee lack of bias.

As far as I understand it, in the U.S., the game is that one needs to make a good impression for a while, but, then, at a certain point, one has tenure, and, in mathematics, if one has not planned finances to depend on huge external income from grants, one can exercise one's own judgement.

The abstract-economic conflict is that there are far more people who'd like such situations than there are situations. How to decide? ...

EDIT: in light of the original questioner's comments below: the premise that grades reflect the seriousness of a student I think more reflects the assumption about what "serious" means. E.g., it is possible to very scrupulously follow instructions, and to pay careful attention to what will optimize one's grade, without caring at all about the subject of a course. And, in many math courses, the "assigned problems" (from traditional textbooks) are substantially a caricature of what the subject truly is. Often, artificial makework, to fulfill the mythical need for "lots of problems". So I am not particularly interested in performance in regard to such.

There is also the implicit premise that "mathematics is obviously objective". In a certain sense, this is relatively true, but the question really should be about good mathematics. That is, (logical) correctness is not the highest virtue, by any means. And not all details are of equal significance, by far.

So, to my mind, capacity to see a larger picture and infer wise actions from it is a very important thing to cultivate, for professional mathematicians. The basic stuff, in undergrad and beginning grad courses (in the U.S.) is so standard that there are many sources, and anyone who's interested can learn it from those sources. In particular, I tell my students to not over-invest in drilling themselves on things that in real life are easy look-ups. Rather, the general "physical" sensibility and context of questions is what one needs to reflect upon... not standard stuff. Knowing what is standard and what is not, not the power to reproduce it on command, is the desired skill.

And, so, again, performance in usual coursework or GRE and such really tells nothing about the future sensibilities of a student. Letters from astute mentors/advisors have a chance to do so.

  • 20
    Which raises the question: What useful predictive information is contained in recommendation letters? And is it predictive of anything besides conformity to prejudices about who should be a mathematician, i.e., self-replication? – Elizabeth Henning Oct 4 '17 at 3:01
  • 2
    @ElizabethHenning, yes, of course, degenerate letters of recommendation may fail to reflect anything more than conformity... "Looks like a mathematician", etc. But GRE and GPA most often test for conformity in certain peculiar ways, too. At the best moments, letters of recommendation can actually circumvent pressures to conform, and biases of various sorts. – paul garrett Oct 4 '17 at 12:27
  • 2
    As in my comment below: In Nature, vol 510, 12 June 2014, pp 303-4, Casey Miller and Keivan Stassen wrote a column "a test that fails", with many stats about the GRE. – paul garrett Oct 4 '17 at 21:23
  • 4
    @paulgarrett GREs/GPAs are irrelevant to my point, which is that a "good" letter makes comparisons to other students who have gone on to succeed in grad school, possibly implicitly including the letter writer himself. And that is the very definition of self-replication, unless there are other independent characteristics which have predictive validity. If so, what are they? – Elizabeth Henning Oct 5 '17 at 0:01
  • 2
    @ElizabethHenning, yes, indeed, if all that a letter does is make comparisons to other students, or to the letter writer, it may perpetuate stereotypes, etc. And, yes, this is common. But there is also the possibility to predict success for a not-stereotypical person due to their talent for, and interest in, mathematics, due to those traits. In fact, for people from unfortunate situations, fighting through them and still having an inexplicable interest in mathematics is a very strong positive signal. Much better than comfortable, coddled kids just continuing their undergrad major. And so on... – paul garrett Oct 5 '17 at 0:15
26

You are correct that the system of letters of recommendation is pretty universally disliked, by the people who need letters, by the people who write letters, and by the people who read the letters.

But the problem is that nobody has come up with a significantly better system. In particular, let me outline a few of the things that don't work:

  • We know, from experience and research, that undergraduates' GRE and GPA scores are poor indicators of success as a graduate student: There appears to be little correlation between GRE scores and the probability that a student succeeds in a graduate program. Rather, personal and interpersonal skills seem to be important determinants whether students succeed in graduate school. There do not seem to be other quantitative measures a selection committee can rely on other than personal references.

  • When applying for postdoc positions and starting tenure track positions, applicants may be 1-3 and 5-15 publications, respectively. There are typically no grants or awards at this level, so publications are the only quantitative measure available. But publications are difficult to assess quantitatively: What you really want is an applicant's ability level, but what you get in a list of publications is a melange that also includes the abilities of coauthors, the question of where the papers were published (which may not always have been the best location), your inability to assess whether the best work of the applicant may still be stuck in preparation or peer review, or whether an applicant does great work but writes so poorly that their manuscripts are too often rejected (something that they may still learn, or that could be rectified with appropriate mentoring and/or coauthors). There is of course also the issue that you're going to have someone as your colleague for the next 30 years, so you want to filter out the jackasses who don't get along with folks. In other words, there are again no good quantitative measures that would assess the qualities you are looking for in applicants.

The only place where I would think that letters of recommendations are pretty pointless is for hires of people at the senior level -- say, ten or more years after their first faculty positions. By that time, there is a sufficiently long track record of publications, citations, grants, speaking invitations, etc, that one can form an objective picture of a candidate from a CV alone. These people are also typically known in their communities (and if they are not, then that's a sign as well).

For all other cases, however, I think we just don't have good "objective" measures to determine whether an applicant for anything is good or not, and so we rely on letters of recommendation.

  • 3
    "We know, from experience and research, that undergraduates' GRE and GPA scores are poor indicators of success as a graduate student". Thats rather interesting. Do you have a source at hand? – Polygnome Oct 4 '17 at 8:52
  • 9
    @Polygnome, having been involved with graduate admissions for 35 years, and occasionally as director of grad studies in math, I've observed about 700 grad students' trajectories, and I'd agree that GRE is not a good predictor, nor undergrad GPA. They simply measure different things than what is needed for graduate studies... – paul garrett Oct 4 '17 at 12:23
  • 4
    @paulgarrett I do not seriously question that this is true. its something common sense dictates. Common sense however is often not an argument. that why is asked for some more sources instead of "some stranger on the web told me". – Polygnome Oct 4 '17 at 14:10
  • 1
    @Polygnome -- I don't recall the exact reference, but here are a number of studies: google.com/… – Wolfgang Bangerth Oct 4 '17 at 14:19
  • 2
    @WolfgangBangerth You posting such links is beyond rude. If you know some good, usable sources, I'm very welcome. Otherwise its quite hard sometimes to vet authors of blog articles and even some journals without being from the field.Google puts out a lot of garbage if you just know how to play the SEO game. – Polygnome Oct 4 '17 at 14:30
10

A brief extra word on top of Wolfgang Bangerth's and paul garrett's excellent answers:

1) Here's a few references as a starting point for discussions of success in graduate school, and admissions:
a) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3923635/
b) https://doi.org/10.1371/journal.pone.0166742
c) https://doi.org/10.1371/journal.pone.0169121

2) The best prediction of future research success is past research success. However, especially for undergraduate research, the "objective" lines on a CV are very noisy measurements. This is partially because there are fewer publications, but more because how credit is given to students (especially undergrad researchers) is widely variable. Examples: if you are 5th author, does that mean you washed bottles or you proposed and did a side experiment that got bundled into a different paper? If you are first author of a paper, did you do the experimental design and writing, or just the data collection?

Letters are not just a pile of flattery - they can describe what the student actually did, how much insight they provided to a project, etc. This part is, in fact, basically the only detailed description of what an undergraduate has actually accomplished themselves in doing research. So it's not surprising that they are useful.

[I'm going to put one minor caveat here. There may be some signs that someone will be successful using an objective test in Mathematics. The strongest example I can think of is the Putnam exam. However, even there, there is a great deal of skepticism that it predicts research ability - since one is problem solving on the six hour scale and the other on the six month scale. See this answer: https://mathoverflow.net/questions/15848/what-to-look-for-in-applicants-to-graduate-programs-in-mathematics More importantly, though strong results are a good sign, weak results may not be a bad one - I have been told there are successful math professors who received zeros on the exam.]

  • 1
    I would be stunned if Putnam or other math contests have any independent predictive validity whatsoever. In particular, they are heavily biased towards non-URM men from professional-class (or at least educationally savvy) families. – Elizabeth Henning Oct 4 '17 at 23:53
  • Hm. This is an interesting point - certainly the demographics of the winning US teams have been biased in that way - but I'm not sure if it's more or less than the usual math grad student population. Re: independence - this is suggesting Putnam basically correlates w/past experience and math enthusiasm? Possibly! – AJK Oct 5 '17 at 2:55
  • I think Putnam probably provides absolutely no information about interest or talent for mathematical research that wouldn't be available in other indicators. So after controlling for these other indicators, a high Putnam score doesn't tell you anything except that someone is a non-URM man, who are many times more likely to have the opportunities necessary to achieve a high Putnam score. – Elizabeth Henning Oct 5 '17 at 22:21
  • I suspect that being genuinely the close to the best at anything is independently predictive of success. So I'd be really surprised if say the Putnam top 10 weren't strongly predictive of research success (and I think the lists at the very top bear this out). That said, I'm not sure whether that'd be more predictive of research success than say a strong math student who'd also been a professional musician or an olympic athlete or a top 10 crossword person. – Noah Snyder Dec 10 '17 at 20:11
2

Here are some things that reference letters provide that cannot be determined from a applicant's "statistics" (i.e. grades, number of publications, etc.) that would play an important role in their graduate admissions.

Reference letters speak to an applicant's character

In my country (Canada) it is generally expected that for a graduate degree you have informally worked out who your supervisor will be before you complete the grad school application. If you can't find a professor willing to supervise you, you generally won't be accepted.

For a professor, accepting a new student is essentially selecting a new colleague. The student is someone that the professor will have to work with for at least several years, so it is important to have some idea beforehand what kind of person the applicant is, and whether they will be a good fit for your research group. Reference letters from previous supervisors/professors/employers can tell the professor whether the person is reliable, courteous, a good team player, a good communicator, etc. Just because someone is brilliant on paper doesn't mean you would enjoy working with them.

Reference letters can provide context for an applicant's "statistics"

Sometimes, through no fault of their own, a person will have a particularly bad year. Personal tragedy, health issues, etc. can all play a role in reducing an applicant's appearance on paper by affecting the number of publications that they could produce or negatively impacting their grades. On the other hand, a person might have experienced one of the above setbacks and was still able to produce impressive statistics. A good reference letter can frame the applicant's "statistics" in a different light for the person reviewing the application.

Side-note: gain respect, not favour

In my experience it is not necessary to come up with "strategies" to "gain the favour" of respected academics. What you want is respect, not favour. The difference is very meaningful, and will be clear to anyone reading your reference letters. The favour of a narcissist (yeah, they exist in any field) can be gained by giving them compliments, but narcissists aren't good referees. You can gain a person's favour by running errands for them or giving them gifts, but again, they won't be a useful reference because they won't really know you by the qualities that make you a good applicant. Gaining respect is more challenging and it will vary from person to person. Some people respect confidence, some respect humility, some respect raw talent, etc. Everyone respects hard work (everyone worth your time anyway). Gaining respect isn't a game where you need to come up with "strategies".

  • I agree; the system works best when both objective measures (which are less prone to bias) and subjective measures (like reference letters, which can address the "intangibles") are used in tandem. – J.R. Oct 4 '17 at 18:01
  • "Reference letters speak to an applicant's character": I don't think so: reference letters give the writer's perception of the applicant character, which can be heavily biased by the writer's character. – Massimo Ortolano Oct 4 '17 at 20:44
  • @MassimoOrtolano While I agree that you will tend to see positive bias in reference letters (the applicant would be silly to pick someone who has a negative opinion of them), a good reference letter provides evidence for the applicants qualities and goes into specifics. When it comes to "character" there really aren't any solid objective measures, which is why I said in my answer that the letter "speaks to" (provides evidence for) the applicant's character. It is up to the reader to judge the objectiveness of the letter and determine how much weight they want to put on it. – CuriousFindings Oct 5 '17 at 11:26
1

These are actually several questions IMO:

Can we stop looking at recommendation letters [which are subjective and biased?]

Not entirely, but certainly their weight can and often is smaller than you seem to believe is universal.

Is the letter of recommendation really an indispensable part of academic application processes?

Nope. I was accepted to my current post-doc in the Netherlands with no recommendation letter. I was asked to provide references in the form of contact details though. And this way is not entirely uncommon (but I have no statistics).

My question is, Can "objective" things like GPAs, test scores, publications alone provide enough information to evaluate the strengths and weaknesses of applicants?

Grade point averages, in my opinion, are definitely not to be relied on heavily - not because of their (better) objectivity, but because they:

  1. Measure something which is not as well-correlated with what you actually want to be assessing
  2. Have non-subjective biases due to institute grading culture, country of origin, etc.
  3. Don't cover people whose background is unusual and who might want to get into research

but for screening the non-exceptional cases, I guess some minimum GPA requirement could be used.

As for relying on publications - either you rely on numbers of publications, which is a bad measure IMHO for multiple reasons which I won't expand on here, or you have to actually look at the contents of the publications, which is probably much better but much more time-consuming.

Bottom line: IMO you can decrease the reliance on recommendation letters but you'll "pay" with an increased effect of other problems and issues.

0

In my experience this is more just a statement that the Prof. knows that student and generally thinks well of him and can recommend the students. I don't think the details matter that much, especially since those are often written by the students anyways. It's basically the same with invitation letters.

Meanwhile, writing letters seems to be a heavy burden of professors.

Is it? Many Profs seem to not have any problem with this and the others just ask for a "draft" which they usually sign. In addition it doesn't take much time to write such a letter and the students maybe worked years in the lab of the Prof. and he shouldn't spend some minutes writing a letter?

Can "objective" things like GPAS, test scores, publications alone provide enough information to evaluate the strengths and weaknesses of applicants?

They can, but they cannot tell you if the student got the support of the Profs or even know them.

For example I applied for a small grant to spend some months in a research group in another country. I talked to the Prof. there, we discussed a possible topic and cooperation and he invited me to his lab. Including a letter of invitation saying that I approached him with several ideas and he is happy to host me and I have his full recommendation shows that I did all of this and he really wants me to join his lab. This is something my publications, honors, grades,... don't show at all.

0

I wanted to add an answer to this question earlier, but as I was writing it, I realized there is no easy answer.

Research is quite complex and it's not easy to predict people's performance based on anything that could be called "objective metric". Certainly someone with perfect GRE scores and GPA=4 would seem like a great candidate for a graduate program. But, the same person could be like me and have some psychological issues that prevent them to do good research. In my case, I like working alone and I lack patience, even after so many years of training myself.

It gets even worse at later stages of career, when people evaluate you based on bibliometrics. Those sound "objective" to them, but an h-index 20 might mean you were just lucky to be working in a very strong group, while the other guy with h-index 5, who won't even get considered for a faculty interview, is close to inventing the warp drive.

Letters of recommendations seem to me a good way to offset some of what those objective metrics tell about people. If you are the reader of such letters, you should look for facts confirming or not your evaluation of the candidate's ability based on what you know from their test scores, published results, etc. Those letters also tell the interviewer how the candidate works with people. They are not sufficient and should be doubled by an interview.

Even with the interview, you still cannot fully predict how the candidate will fare in the future. You might get a stellar candidate, but for whatever reason they end up working with someone they can't get along with and end up wasting years in graduate school or as postdocs without any serious result. You could also get a regular candidate who ends up working with a group in which they fit well and their career takes off. I have seen plenty of my "average" colleagues becoming great researchers simply because they were wise or lucky to choose the right people to work with. I'm not saying they didn't put in the effort, I'm saying that others with better grades and test scores didn't fare as well simply because they weren't in the right environment.

There is another reason why people need letters of recommendations. There are people who simply don't fit in the system, but would do well as researchers. I knew a student who was building robots. He didn't have any guidance, he learned everything from books and forums. His GPA was slightly above what was needed to pass, SAT was abysmal, so no normal undergraduate program would consider him. But he had a great recommendation letter from his science teacher who knew about his robots, so he got into an undergraduate program. Now he's doing well as a graduate student, and has quite a research record.

0

The problem becomes clearer when you compare what someone sifting students will probably be looking for.

At postgraduate level, you want to know what their prospects are at research. But many graduates haven't done formal research as such, and is really hard to design a grade system to rate their prospects as a future researcher.

They will also have to move much more to self development and be able to collaborate with others _- the grade system might rate how much they know, but not what kind of person they are like to work with.

Postgrads often do some level of teaching in the field. In an office they might mentor or supervise others (in some cases). Again, grades don't really tell the recruiter how they are likely to be as a person (not just in terms of technical knowledge) at this.

For similar reasons a whole bunch of other skills that are fundamental to deciding who to accept, just cannot be assessed on grading, and even assessing by interview performance is often missing key information.

The same goes for commercial as well as research roles - businesses will want to reduce risks of a problem, and the information let's them improve their chances. So a bias favouring candidates with such information will often exist.

The human perspective, by someone else, independent, who knows the person, is still extremely valuable and key to many decisions of this kind.

-3

Let me put it this way. After reading your question I would never recommend you for a job at any place I work. Why you ask? Because you come off as arrogant and naive. Naive people I can work with, arrogant on the other hand is not worth it. One cannot tell this from quantitative measures like the GRE.

No matter how smart or skilled you are, I would not want to work with you. No person can work in a vacuum they need assistance be it grant writing, IT support, accounting, payroll or any number of other things. You can't get things done without assistance and managers (Professors) realize this. If you drive away them or their staff (which they have worked very hard to build) it does not matter how much you think you can do, it's not worth working with you.

  • 3
    I don't see how this answers the question. – Massimo Ortolano Oct 4 '17 at 20:38
  • @MassimoOrtolano OK I'll make it explicit: "No people will never stop looking for qualitative assessments because they provide value" – Sam Oct 4 '17 at 20:46
-3

The PageRank algorithm ranks a page as more authoritative if many pages link to it, and a page as more relevant if an authoritative page links to it.

On Stackexchange, users earn a reputation from the quality of their answers, as conferred by fellow users, which lends weight to their subsequent answers.

In academia, professors earn respect by the quality of their work, and the testimony of a respected expert in the field transfers respect onto whoever they recommend.

I think it's just human nature to be swayed by anecdotal arguments from authority more than statistical ones. To wit, an anecdote: I've submitted recommendations to a system where I had to check boxes for how unique an individual was in several criteria. The levels of uniqueness are based on how frequently I have encountered students at their level ("top 10% each year," or "top 2 or 3 in my career", etc). The criteria are things like "oral communication," "independent thinking," "personal responsibility", "leadership", etc. This could result in some statistical measure of the quality of an applicant.

I hate these matrices! I might have an opinion on a few of the criteria, but on others I really couldn't say. And I've taught maybe 5000 students in my career; asking me to arrange them in increasing order of ability makes me tired just thinking about it. So I'd much rather upload a letter, in which I give quantitative data on the student's performance, and personal qualities which show aptitude for whatever they're applying to.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.