The GRE: Why does this still exist?

Question

I did my undergraduate degree in the US and am heading to graduate school here in less than a month, so I myself have taken the Graduate Record Examinations (the general as well as two subjects tests) and I guess it always just seemed as a sort of un-avoidable formality that nothing could be done about, and so I just took it and got it over with.

In the intervening year between undergrad and the start of my Ph.D. work, however, I traveled overseas to Cambridge where I found that I was quite mistaken: the GRE is very, very avoidable. The solution is simple: don't apply to universities in the US.

By the time I had arrived there, I had already gotten it over with myself, but for most of my peers there, this was not the case, and quite a few of them had simply decided to not even bother applying to the US because of the inconvenience that comes along with that in the form of the GRE.

This made me wonder: are admissions committees at US universities aware of the number of highly qualified candidates they miss out on because of the GRE?

I could understand being willing to miss out on the potential recruitment of these students if the GRE were a significant part of one's application, but I have yet to find any US professor tell me that the GRE scores are weighted highly when it comes to making admissions decisions (perhaps I just haven't asked around enough?). In fact, I've often been told it's the least important factor when deciding whether someone should be admitted. (Indeed, my impression is that the general GRE is more or less a joke and only serves as a convenient way of tossing out applicants who would have been found un-qualified for other reasons.)

Putting aside for a moment the issue of those who decide not to apply to US universities, let's consider the inconvenience faced by those who do. Once again, if you're from the US, I can imagine simply not being aware of this (I know I wasn't), but I now know of several people who have had to fly (sometimes the flights have even been inter-continental!) in order to sit to take a GRE test. And even for those who don't (like probably most of us in the US), there is the ridiculous price: almost $200 for the general and an extra $150 per subject test. I was under the impression that admissions committees encourage people from all backgrounds to apply, rich or poor, but how can they honestly expect this to happen if even those who don't have to fly have to shell out anywhere from $300-$500 in addition to the application fee? (I personally find it a bit nuts that these tests cost several times more than the application itself.)

So, could somebody please explain to me why we still require students to take these things? Do they really add information about the applicant and their abilities that could not be found out any other way?

Answer 1

Empirical evidence on the relationship between GRE scores and post-graduate performance

There is a massive meta-analysis by Kuncel et al (2001) that empirically evaluates the correlations between various aspects of the GRE with multiple post-graduate performance criteria.

Based on hundreds of studies and thousands of participants, GRE shows reasonable correlations with post-graduate GPA (i.e., around observed r = .21 to r=.43). Similar correlations emerged between GRE and faculty ratings of the student.

Correlations of GRE with research productivity and publication citation counts were smaller, but still positive. This is not surprising given that these are more distal outcomes and there are many non-academic reasons why people may or may not pursue an academic career or have publishable results.

The authors thus concluded that GRE was a valid predictor of a wide range of graduate outcomes. They also noted that "subject tests tended to be better predictors than the verbal, quantitative, and analytical tests."

In general, selection decisions are assisted by standardisation, and a big part of academic achievement involves measuring baseline ability. Thus, the GRE combines both standardisation and competence measurement.

Update 2023: A new meta-analysis was published by Feldon et al (2023) that reviews GRE correlates with academic outcomes. They report correlations between GRE and GPA of .24 (total), .20 (verbal), .17 (quantitative), and .21 (analytical).

Response to comments

There have been a few points made in other answers and in comments, which I'll comment on here:

• Conflict of interest: @msw wrote that they "consider the cited paper to be junk because ETS provided all available data". In general, I don't find the results in the meta-analysis surprising. Most tests like the GRE tend to have fairly strong correlations with general cognitive ability. It is well established through thousands of independent studies that IQ scores correlate fairly well with both school grades and job performance (i.e., in the r=.50 range; see Neisser et al 1996 for a field consensus review). The results show correlations less than .50, but that's not surprising given some of the issues around standardisation, practice, range restriction, domain specificity and so on.
• Belief there is no correlation based on personal observation: Note that if the correlation is around .20, that means that 4% of variance has been explained. That leaves a huge amount of variance in performance to still be explained. It would not be surprising to meet many people that did well on GRE and poor in graduate school or vice versa. Thus, it is problematic to rely on personal experience when it comes to evaluating the validity of tests where such correlations are likely to be only modest.
• Small correlations are useful: While a .20 correlation is small, it can still help make selection decisions. In particular, when evaluating the suitability of selection tools, you need to contrast the validity of a given tool with other available tools (e.g., interviews, GPA, references, and so on). I'm not as familiar with results in the graduate selection domain, but certainly in the employee selection domain, which is quite analogous, cognitive ability tests tend to correlate more highly than interviews, references and so on (for a comprehensive meta analysis of employee selection, see Schmidt and Hunter, 1998). That said, the best selection decisions are typically obtained by integrating multiple selection tools. Furthermore, the small correlation also should highlight to individuals who score poorly on the GRE that it is not that predictive, and therefore it shouldn't discourage an individual from pursing post-graduate study.
• Does training invalidate the GRE?: @user8134 wrote "many people significantly improve their GRE scores by taking courses with prepping companies like Kaplan. So it seems GRE test does not measure any intrinsic ability/talent necessary for grad. school." I agree that individual differences in training and preparation for the GRE may influence test scores. That said, if you characterise test scores to be determined by true ability, training, and error variance, then I would expect that true ability would remain the much larger source of explanation in test scores. This is based on general observations about testing for ability based assessment. In general, the degree to which training is an issue would depend on how much the test materials teach to the specific test. Overall, I would assume that this would reduce the potential validity of GRE, but that the GRE would still be useful. Furthermore, training and nuisance factors can be used to do better on many selection instruments. For example, people can be coached on how to frame their CV or how to answer questions in interviews. Such training is potentially a source of error variance, but it doesn't invalidate CVs and interviews completely.
• Why would anyone care about post-graduate GPA? (@JeffE) @JeffE further notes " in PhD programs, the only thing that really matters is the student's research output.". Some post-graduate courses include meaningful graded coursework and others don't. For the courses that do include meaningful coursework, then such coursework provides a more standardised way of measuring post-graduate performance. So the validity of GRE in predicting such outcomes is not surprising. And thus, presumably we could generalise this to being indicative of how people perform in less standardised aspects of post-graduate performance. Of course, there's an inferential leap here, but in general, performance in related domains tend to correlate (e.g., coursework in mathematics with research performance in mathematics); it's not perfect, but it's still a positive correlation. Furthermore, if the validation study includes some post-graduate coursework where everyone gets top marks and such data is mixed with studies where grades are valid measures of performance, this would only serve to attenuate the observed correlation. Thus, this would suggest that the correlation between GRE and post-graduate GPA is higher than reported by the meta-analysis. Also, the meta-analysis does report correlations with research output and they are weaker but still positive. It also reports correlations with supervisor ratings.
• Better alternatives to GRE: None of my comments above are necessarily advocating the use of the GRE. Developing an effective selection and recruitment system whether it be for employment or post-graduate admission is a complex task. That said, most post-graduate selection systems would want to get a reliable and valid measure of academic aptitude. GRE, IQ tests, other ability tests, undergraduate GPA, all have reasonably validity evidence. And in general standardisation and efficiency are important. So, for example, administering your own selection tools takes more time, whereas taking pre-existing measures like GPA and GRE is more efficient.
• Ethics of requiring applicants pay money to complete GRE: Several people are critical of the GRE on the basis that it costs several hundred dollars to complete. I think that this is a perfectly legitimate question, but that the question of predictive validity can be answered separately. Such a fee could potentially discriminate against low income applicants. That said, presumably the fee in comparison to forgone wages associated with completing a post-graduate degree is fairly small.

References

• Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance. Psychological bulletin, 127(1), 162. PDF
• Neisser, U., Boodoo, G., Bouchard Jr, T. J., Boykin, A. W., Brody, N., Ceci, S. J., ... & Urbina, S. (1996). Intelligence: Knowns and unknowns. American psychologist, 51(2), 77. PDF
• Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological bulletin, 124(2), 262. PDF
• David F. Feldon, Kaylee Litson, Brinleigh Cahoon, Zhang Feng, Andrew Walker & Colby Tofel-Grehl (2023) The Predictive Validity of the GRE Across Graduate Outcomes: A Meta-Analysis of Trends Over Time, The Journal of Higher Education, DOI: https://doi.org/10.1080/00221546.2023.2187177

Answer 2

This is a good question. I have not been a fan of the GRE for 20+ years, although (through whatever luck I had a good-enough number on it myself that it didn't harm me...) many are. Having been on admissions committees and very much involved with graduate programs for 30+ years, I've had ample opportunity to see the (non-) correlation of success in graduate mathematics with GRE subject-test scores. (The other parts are often useful as tests of English fluency, mainly.)

Of course, the world would be a simpler place if GRE subject-test scores really could show talent for higher mathematics. We note that the Educational Testing Service (in NJ, that makes the GRE and other stuff) is a for-profit that has a vested interest in maintaining its products' apparent importance.

The cost is unfortunate, and certainly discriminates against people whose currency doesn't compete well with USD.

The vaguely useful bit of information provided by GRE is that, well, yes, it is the only thing that most applicants will have done, thus, if one insists on "simple comparisons", it is the only thing that allows that. It is clear that comparison of GPAs is even more pointless.

I have known admissions committees that simply ranked applicants by GRE subject test score. There! Done! :)

No, I do not care very much about GRE numbers, but in a way I am glad that some admissions committees do, in a fashion to the way that under-valued stocks are good investment values.

I do think that the elite graduate programs use GRE subject test as a convenient filter, because it selects somewhat for "quickness/cleverness", and they can afford to "lose" some prospects, because they have so many who are "quick/clever".

In the U.S., having a GRE subject test score is also a sign of awareness that people expect you to take it. Thus, it doesn't matter so much what one's score is, but that one _is_aware_ ... even if it is only of "expectations".

But, in summary, no, I see no point in it. But there are economic incentives for ETS to keep making money. And a great number of admissions committees in math have personal predilections that lead them to be fond of (over-) simple numerical quantification, so... there-we-are.

Edit: As @msw observes in a comment, indeed, if GRE measured significant academic achievement cumulative over several years... it would be odd that one could usefully do the "prep" courses ETS provides. :)

Yes, performance on GRE probably is a good indicator of how well a kid can do on a multi-hour, timed, multiple-choice test, etc. Yes, if we make subsequent coursework resemble this (!?) then we give the GRE predictive power. No, I do not recommend making everything multiple-choice! But, amazingly, some people do believe that this could be done, and purportedly save us all a lot of work.

Sure, these things measure something, and produce numbers that can be manipulated. There are people who are inexorably drawn to the possibility of making final decisions in those terms, even when the significance of the numbers is unclear. Meanwhile, reading letters of recommendation and personal statements is obviously not easily quantifiable. Of course!

If it were really the case that "standardized testing" could tell what its promoters like to insinuate, it would be convenient, indeed. But, again, some decades of experience indicate that these tests do not indicate whether or not people can sustain interest over 4+ years, work hard for 4+ years, continue to develop scientific sensibility, and so on. And the latter issues prove to be vastly more important for completion of a PhD. In direct observation of about 700+ grad students, I'd estimate that fewer than 20 dropped out or failed due to lack of intellectual capacity or lack of prior knowledge. Rather, loss of interest in the subject, or personal issues (mental/physical health) dominate. This "sample" of mine includes a very wide range of GRE scores and even GPA.

More anecdotally, several specific examples stick in my mind, of very low percentile on GRE subject test (bottom 10 percent or smaller...) but exceptional achievement in coursework, prelims, and thesis work. These peoples' potential was easily visible in letters of recommendation and personal statement.

Answer 3

In my department, GRE scores are used mostly as calibration for students who have good grades but are from universities/programs that we are not familiar with and whose quality we are therefore not sure about. For students with good credentials from strong programs the GRE is pretty much irrelevant.

Answer 4

A couple of issues:

Re Mr. Anglim's comment, a suspect issue re. any correlation between GRE scores and success in grad. school is that many people significantly improve their GRE scores by taking courses with prepping companies like Kaplan. So it seems GRE test does not measure any intrinsic ability/talent necessary for grad. school.

Another issue is that the GRE is a form of forced labor: ETS uses one of the sections in the exam (just which section is unknown to the test-taker) as data for future exams, i.e., the section is not counted for the score of the test. So one is expected , basically, to work for ETS for free, producing high-quality data they would have to pay a lot for, or may not be able to produce themselves. GRE also puts out books to prepare for the exams, which cost above $20 each.

Now, ETS could find a way around this by asking, say, "There may be a section in this test which we use as data for future exams: if there is one, would you be willing to take it, or do you prefer to skip it?". This - asking you to work for them for free - is unethical, IMO, if not illegal. And ETS' BS response to this (I called them) is to tell you: "Well, if you disagree, don't take the test." The problem is some programs require you to take it in order to apply for their grad. programs; ETS is the Frank Burns of testing.

Answer 5

@Jonathan, if you were to ever come to the other side, you will see a great number of glowing applications from top students of the top university in Bolivia or Madagascar. How do they fare compared to the top student of the top university of Idaho? Or a top student from a mediocre university in Massachusetts? If you admit grad students planning to use them as TAs, how do you know that this student from Nepal will be understood by your students in Tennessee, who heartily laugh at both New York and Californian accents?

The GRE fills the role of such a filter, and as such is the cheapest, easiest to use tool available to graduate schools in the US. If you can get any UNESCO money to design and implement a version of it that would be free to international applicants, maybe you can get a Nobels fredspris for your efforts. (This is not such an unrealistic idea as it sounds: an alternative, freely available operating system known as Linux has been developed by enthusiasts, and in many dimensions has replaced UNIX.)

Answer 6

In a word, standardization.

I once asked this question to a colleague, and I appreciated the response I received.

Essentially, the faculty member told me, "What I like about the GRE is that it's the only way I can compare apples to apples."

He went on to explain, "How can I compare a 3.2 GPA at University X with a 3.7 GPA at University Y? I can't. But, at least with GRE scores, I can compare the two students on an even playing field."

My retort was that a high GRE score isn't necessarily a good indicator of potential in graduate school, although it might be a good indicator to determine the amount of time a prospective student spent preparing for the test. (I got a lot of milage out of my How to Ace the GRE practice book.)

He readily agreed, and assured me that it's just one piece of the puzzle. An admissions office only has so much to go on: a transcript, a GRE score, and perhaps a "Why I want to go to graduate school" cover letter.

We could throw out the GRE, but then there would be that much less information to base admissions decisions on.

My gut tells me that the negatives include the expense and hassle for the applicant, and a limited ability to predict how the student will actually perform. But I must admit my colleague had a point with his "apples to apples" perspective.

Answer 7

Explains the quality of our PhDs, doesn't it. I have personally found the American education system far more wrapped up in silly and actually dangerous formalities than schools in Europe, generally speaking. I was surprised to find this out first hand because the common consensus is that Europe suffers from a great deal of pointless formality that the US does not. The problem is that it's a category mistake: elaborate bureaucratic formalities: yes; greater social formality: sometimes, depends; educational formality, not necessarily. Also, there is a difference between formality and rigidity. While titles are more important in Europe when addressing one another, they actually can enforce a very healthy relationship between the student and the professor which enables a healthy flexibility and informality to emerge. The casual "buddy" culture of the US is actually too chummy which encourages disrespect and I think the reaction to it is a certain kind of seething rigidity that manifests in the student-professor relationship which is unhealthy and domineering many times. But I digress. When I experienced American education first hand, it seemed infantile in its execution and the GREs belong to this set of things which contribute to the culture of pettiness (other things include the "publish or perish" doctrine which has resulted in the explosion of BS and CV padding nonsense and the "career academic" (hopefully that translates into English properly). While there are entrance exams in Europe, graduate school acceptance is according to other criteria. It's not always optimal, but certainly less irritating than this test taking nonsense.

Btw, I find the correlations mentioned above to be exactly the problem. They make the same mistake of drawing rigid and often erroneous inferences from a very poor selection of data.

I find that university education everywhere have been ruined by the pressure to put everyone through it. What you have are glorified trade schools in most cases with millions of applicants pushing through as if through a military entrance exam where narrow indicators sacrifice complexity for measures of simple routine. It's highly corporate and prone to the rat race. Graduate school should be the last place where this kind of pettiness manifests, and yet...

Summary: GREs are a result of the culture, of modern (American) university culture like the silly innovations of the German university which sought to label and shelve every person for particular slots in the machine of the the Reich.