Is it ethical to apply different criteria for graduate admissions based on country of undergraduate study?

Question

We have undertaken a small statistical study of M.S. students in our department, including their application information and their eventual performance in our program. The goal is to develop criteria for making admissions decisions for new applicants based on their likelihood of success in our program (as predicted by the performance of recent, similar students). I won't get into the details of the methodology.

One outcome of this study was that different attributes predict success for students with undergrad degrees from different countries. For example, for applicants to the program from schools in country X, but not schools in country Y, undergrad GPA is correlated with the students' GPA in our program; for schools in country Y, but not X, GRE scores are correlated with the student's GPA in our program. (I'm simplifying a lot here.) A "toy" example I just completely made up to illustrate is shown below:

(of course, for real applicants the criteria and the relationships are more complicated)

There are many possible reasons for this: for example, we could think that the grading system is more consistent in X so undergrad GPA is a better predictor there, and in X students study "for the exam" so the GRE becomes virtually meaningless as a measure of knowledge. I could speculate, but I don't think it would be helpful. The bottom line is, we find that different factors predict student success among different populations.

Therefore, if we wanted to admit students based on their likelihood of achieving a certain GPA in our program, we would apply an undergrad GPA cutoff for students from X, and a GRE threshold for students from Y. (Again, this is vastly simplified from the criteria our study actually suggested.)

Is it fair to apply different criteria to students with undergrad degrees from different countries in admissions decisions?

Does the answer change if this would significantly skew the admissions decisions in favor of a particular country of undergrad study (because statistically, applicants to our program whose undergrad degrees are from X have done much better than those with undergrad degrees from Y).

My concern is that we're effectively saying, "Students with undergrad degrees from Y with a GRE score < T will be rejected, but students with undergrad degrees from X with GRE < T may still be considered for admission (pending other criteria)."

On the other hand, if we ignore these statistics and reject students with undergrad degrees from X with low GRE scores, we are rejecting applicants even though we have no valid reason to believe that they won't do well in our program.

To those who doubt the results of the study:

• When I say the study is "small" I don't mean it isn't statistically significant - just that it was not designed to be generalizable beyond our applicant pool. (This is the same reason why I won't give too many details about the study - I don't want anybody to read it and try to generalize from our results.)
• As we know, the sample size is not the only factor that determines whether a given effect is significant. We found that the results are significant, given the sample size.
• The results also seem "sane" (which of course is subjective). It's not unexpected that undergraduate grading standards (and the standard-ness of grading standards) differ by country; or that different educational systems and cultures prepare students differently for standardized exams like the GRE and TOEFL; etc. The specifics of the results (i.e., which criteria are good predictors for which undergrad country) are consistent with what students who studied in those countries have told us about grading standards, student culture, and exam prep. So, we really have no reason to doubt them.

Answer 1

You asked an ethical question and got a lot of scientific, legal, and political answers.

Ethically, you should:

1. Do your best to admit people based on merit.

2. Be open about your policies.

3. Make a serious and competent effort to ensure that your statistical methods are valid logically and empirically, and that they can withstand scrutiny from people who are experts in the field.

4. Carefully consider the historical legacy of racism, colonialism, and nationalism, and work hard to make sure that you aren't inadvertently reinforcing this legacy.

Your behavior so far is probably far more ethical than that of most people in your position, since most such people are probably secretive about their practices (you publicly asked for advice) and probably apply various heuristics without carefully considering whether those heuristics could withstand professional scrutiny.

You might want to expand this small, informal, unpublished, nonprofessional study into something more serious and systematic, done by people who have expertise in psychometrics and the (very difficult) methodological issues of the social sciences.

You are unfortunately operating within real-world constraints imposed by (1) the existence of countries where GRE scores are fraudulent, and (2) the existence of countries with such poor undergraduate education that undergraduate GPAs don't mean much.

Answer 2

I'm going to advise against.

• You say your sample size is small. And honestly, unless you hired a professional statistician, I have doubt the analysis was carried out properly or shows exactly what you think it shows. Good statistics are hard to do.
• Assuming my google results are truthful, National Origin is a Protected Class in the US. You can't discriminate against people because of it. I assume this is an american institution as you talk about GRE, but similar applies in other places. EDIT: OP has since clarified a misunderstanding of mine - the judgement is based not on the student's country of origin, but the country of the school they attended, which makes me less certain this point still applies.
• You're department is actually, seriously proposing to say to people "Sorry, while you have the same official qualifications as another candidate we accepted, we're rejecting you because of your country of origin"? I mean, really? Whoa. Just stop and play back how this kind of justification would sound in almost any other circumstance. Think about how the press will make it sound when they (inevitably) get wind of it.

SUBSEQUENT EDIT

So, this question interested me, and I've bounced it off a couple of other people in my lab. Their main concern seemed to be that grouping by country was just to coarse a measurement, as every country has its own mix of good and bad schools. The basic idea, though, didn't seem to elicit the same gut reaction as it did from me.

Moreover, I've googled and found at least two examples of courses that vary their requirements based on country of origin, so there is precedent:

https://www.auckland.ac.nz/en/for/international-students/is-entry-requirements/is-minimum-overseas-entry-requirements.html

http://sydney.edu.au/business/futurestudents/postgraduate_study/pg_coursework_studies#app_req

So maybe I've got this wrong, and should just be ignored. At any rate I'm no longer sure it's quite so clear cut.

Answer 3

I was waiting and hoping that @JeffE would expand his comment, since I share the opinion he expressed there, and moreover I think he is far more qualified to give advice on the subject. (Quite possibly, if he decides to expand his comment to an answer that has better facts and arguments than what I am about to offer, I will delete my answer).

DISCLAIMER: I do not know either where you are, what the laws are like there, or if your final decision can have any legal repercussions. I advise seeking legal counsel for any questions of that sort.

I think it if fine to have different criteria for students that obtained their undergrads in different countries (and/or in different institutions), as long as you can ensure the criteria are fair.

Now, how to approach ensuring that the criteria are fair, I have no idea, but I definitely know that if in some countries the GPA is indicative of a person who has potential as a researcher, in some countries it is simply not (I for example did my Masters in Croatia, and I know both people with around 3/5 GPA that made excellent PhD candidates and people with a perfect 5/5 GPA that I do not think would be capable of much independent thinking that were happy to take jobs where they have strictly defined output they have to produce and no research to speak of).

I have two different examples of using "different" criteria in the admission process, but unfortunately both of them are for the wrong level of study:

• First one is about undergrad admissions to my former Computer Science Faculty* in Croatia. While the big majority of the candidates take a standardized test and are admitted based on that, a small amount of students (maybe 1%, maybe even much less) are invited.

  These invitations work as such: the Faculty regularly does a continuous performance study on the previous invitees according to their High schools, as well as (I assume), overall performance of students from different High schools. Each year, a number of invites is extended to High schools, who can then "award" their students with those invites.

  Based on the performance study and possibly High school size, some schools get a larger number of invites (up to 5, I think), while some get only 1 invite or even none.

  And, actually, I think it's fair. E.g. the strong mathematical High schools will get the most invites (those are the schools "prepping" their students for technical studies after all), strong general schools would get some, and weak schools would get none. And still everything would be re-evaluated year after year. Also, additionally, nobody still loses their fair chance to enroll: this "invited" students make less than 1% of the enrolled students, while everybody can still take the standardized test.

• The second example is about the interview for a PhD position / pre-PhD internship.

  Recently, a permanent professor from my lab started looking for a person to hire for an internship with a strong chance of offering them a PhD upon the completion of the internship. I mentioned that my ex supervisor used to supervise people with a similar profile to what was required, and that I could ask him weather he has somebody interested (and good enough) to apply.

  When he received the preliminary application documentation, he asked me to comment on the profile because he was not familiar with the Croatian University system at all. I said that good grades were usually indication of a good student, but bad grades do not have to be an indication of a bad student.

  There was also some other points in the CV adding value to his application, not directly obvious to somebody non-Croatian. After talking to an ex post-doc of his (who worked a bit with the applicant), the professor decided to interview with the student.

  A few days later, he told me that based on the interview, he offered the applicant a position. He also told me that he would most probably not interview a person with that profile, if not for what me and his Croatian ex-post doc said about the "interpretation" of the profile.

I agree with both these cases. The ("absolute") criteria is indeed not the same for everybody. But still, in both these cases, the goal was to be fair, to base the decision on the applicants abilities, and finally to judge the applicants abilities and potential "on the same scale" for all applicants, just based on different information that was available, finally causing different "absolute" criteria.

^{*In Croatia, the Universities are not wholesome entities, and all the administrative decisions are made on the level of the Faculty. There is no identification of students with the Uni in Croatia; if you ask a Croatian studying "at home" where he is studying, he is going to provide the information about his Faculty.}

---

ADDITION. I wanted to add that, despite what any statistical research might show, if students from different countries have the same, internationally standardized tests, then I think making a different criterion would be wrong.

Maybe, for the country X, those tests are not good indicator because students get prepped exactly for those types of exams, but there just might be a few students who learned the material in the "proper way" (as do the majority of students from the country Y), who might get rejected for no real reason.

So, bottom line, I think

• studying, and then declaring your own criteria to interpret different (non-standardized) national grading schemes is fine (even if they use the same scale "on the outside" -- e.g. two countries which both have a national standard grading schemes on a scale 1 to 5)

• but, if a criterion is based on an internationally standardized exam, that same criterion should be used with everybody where applicable (e.g. everybody who took that standardized exam).

  As a compromise-suggestion, I think it would be fine to say that you accept all students from everywhere if their score is extremely high (e.g. >95%), and all the students with medium-to-high scores (e.g. between 75% and 95%) will be considered based on additional criteria (where you can introduce personal interviews, standardized test from your institution, personal research statement + recommendation letters)...

Edit2: This opinion was supported when I talked to a person with a background in law (not in academia thou).:
The final verdict was: interpreting criteria that are initially different (e.g. GPA) differently is OK, but if something is standardized on an international level, it would only be OK to either take it in to account equally for everybody, or not at all. (purposefully using OK instead of "allowed" or "lawful" since I don't know the laws at your place). With a note that, unfortunately, What is lawful and what is just and fair is sadly not always the same.

Answer 4

I don't think it has to be bad. GRE is a measurement of how good you are at doing exams, and not necessarily the most relevant kind to your graduate work. Some universities and education systems are very good at making people excel at set fixed format exams; but they don't teach them how to think by themselves and be creative.

In my opinion, GRE is a very biased measurement in itself, and it is not very well designed (just look at the ridiculous maths part). Also, it is based on a lot of fairly simple exercises, whereas in some universities, we are used to completely different kind of exam: a few long and complicated exercises, and perhaps don't have the ability to work fast enough in simple and repetitive tasks.

Answer 5

I think the underlying questions are (ought to be) first, whether it is fair to reject a student because you cannot accurately measure their potential to do well in the program; and second, if you can perform equally accurate measurements on two groups as long as you use different criteria, is it okay to do so?

Answering the second question first: I think the answer is "yes". The reason is made apparent by advocating the alternate view: should you take students who are likely to do poorly simply because you fail to apply a more sophisticated measurement? That seems pretty boneheaded, and not very fair to the best students. You should leave the analysis up to computers, as they are good at this sort of thing and impartial, but it's a good idea if you can implement it. Don't forget, though, that data on whether a score distinguishes students within your program does not tell you about whether it helped you reject students that didn't make it in! So you should be skeptical but open to the idea if that's where the data leads.

The first question could be rephrased as: "if I know I'm not getting a good sense of some students, can I just ignore them all?". And that guides my advice there: no, that is not fair. Find a way to do your job better--to get more information so that you can get a good sense of these students, or just accept that you would rather make mistakes in acceptance than find a way to do better on your predictions.

Putting these two together: if you have equal statistical power across groups when subdividing one population, then great! Use the information. But if you end up with one group better-measured than the other, you should only apply different criteria if you can show that students who are great and do all the right things in their less-measured context still have a shot at being admitted.

(Also, simple thresholds for individual scores, e.g. "We only take people with GREs above X" are rarely a smart way to run admissions unless those thresholds are set amazingly low. You should be thresholding an overall score to get the top N students, so you'd be applying different weights for GRE in one context vs. another.)

Answer 6

I would argue that you are interpreting your data incorrectly and using the results in an unethical and discriminatory way. Your model is not identifying groups of individuals who should not be accepted, but rather groups of individuals that require additional support so that they can reach the full potential predicted by their past performance.

Consider the following example: For a variety of reasons that have nothing to do with ability or commitment, women have historically been less likely to become full professors in STEM fields than men. To reject female applicants base on a model that captures this historical effect is completely unethical. What the model would show is that female applicants need additional support to maximize their potential (e.g., flexible working hours and mentoring).

Answer 7

Does the answer change if this would significantly skew the admissions decisions in favor of a particular country of undergrad study (because statistically, applicants to our program whose undergrad degrees are from X have done much better than those with undergrad degrees from Y).

My concern is that we're effectively saying, "Students with undergrad degrees from Y with a GRE score < T will be rejected, but students with undergrad degrees from X with GRE < T may still be considered for admission (pending other criteria)."

To my ears this sounds analogous (with certain exaggeration) to: "we only want rich, white men from successful parents that could afford to send them to good schools through out their childhoods."

I realise that this is not what you intend but you got to realise that having different criteria based on the country of undergrad studies is unethical, biased and I can't see how you could implement something like that without having major headaches. You can, by all means, have different criteria for an individual (for instance by having an interview) but if you clump up an entire population based on some statistics, which you can't elaborate on for some reason, I call that discrimination.

Bottom line is, you cannot evaluate the chances of an individual being successful or not, without giving that person a fair chance. If you deem a particular GPA/GRE score to be "good enough" for your graduate program, all applicants that satisfy that criteria should be considered good enough, regardless where they come from. Any additional selection criteria could be justified only if it warrants additional information, such as TOEFL score for those who don't have English as a mother tongue.

One other option would be to calculate "success rate coefficients", something like a multiplicative factor for the GPA/GRE score for applicants from different undergrad institutions, which could in theory be a fair assessment, but practically unfeasible considering the number of possible institutions involved.

Another alternative would be to devise a test for your institution, that you consider to be more fair than using GPA or GRE score. But even that, judging by your comments, is not an acceptable solution. Honestly it sounds a bit like you just want confirmation of some sort.

Answer 8

I would "massage" this problem by including a dummy statistical variable for the country. E.g. 0 for the country where grades are more important, and 1 for the country where GREs are more important. You could almost separate the applications into two piles this way, and tackle one pile at a time.

If asked about it, I would answer that neither grades nor GREs alone are dispositive, and they both have have greater explanatory power when you introduce the third (country) variable into the equation.

As far as I'm concerned, it is ethical to use any combination of statistical variables that satisfactorily explain performance, and to strive for "best fit" (statistically).