Take the 2-minute tour ×
Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It's 100% free, no registration required.

We have undertaken a small statistical study of M.S. students in our department, including their application information and their eventual performance in our program. The goal is to develop criteria for making admissions decisions for new applicants based on their likelihood of success in our program (as predicted by the performance of recent, similar students). I won't get into the details of the methodology.

One outcome of this study was that different attributes predict success for students with undergrad degrees from different countries. For example, for applicants to the program from schools in country X, but not schools in country Y, undergrad GPA is correlated with the students' GPA in our program; for schools in country Y, but not X, GRE scores are correlated with the student's GPA in our program. (I'm simplifying a lot here.) A "toy" example I just completely made up to illustrate is shown below:

enter image description here

(of course, for real applicants the criteria and the relationships are more complicated)

There are many possible reasons for this: for example, we could think that the grading system is more consistent in X so undergrad GPA is a better predictor there, and in X students study "for the exam" so the GRE becomes virtually meaningless as a measure of knowledge. I could speculate, but I don't think it would be helpful. The bottom line is, we find that different factors predict student success among different populations.

Therefore, if we wanted to admit students based on their likelihood of achieving a certain GPA in our program, we would apply an undergrad GPA cutoff for students from X, and a GRE threshold for students from Y. (Again, this is vastly simplified from the criteria our study actually suggested.)

Is it fair to apply different criteria to students with undergrad degrees from different countries in admissions decisions?

Does the answer change if this would significantly skew the admissions decisions in favor of a particular country of undergrad study (because statistically, applicants to our program whose undergrad degrees are from X have done much better than those with undergrad degrees from Y).

My concern is that we're effectively saying, "Students with undergrad degrees from Y with a GRE score < T will be rejected, but students with undergrad degrees from X with GRE < T may still be considered for admission (pending other criteria)."

On the other hand, if we ignore these statistics and reject students with undergrad degrees from X with low GRE scores, we are rejecting applicants even though we have no valid reason to believe that they won't do well in our program.

To those who doubt the results of the study:

  • When I say the study is "small" I don't mean it isn't statistically significant - just that it was not designed to be generalizable beyond our applicant pool. (This is the same reason why I won't give too many details about the study - I don't want anybody to read it and try to generalize from our results.)
  • As we know, the sample size is not the only factor that determines whether a given effect is significant. We found that the results are significant, given the sample size.
  • The results also seem "sane" (which of course is subjective). It's not unexpected that undergraduate grading standards (and the standard-ness of grading standards) differ by country; or that different educational systems and cultures prepare students differently for standardized exams like the GRE and TOEFL; etc. The specifics of the results (i.e., which criteria are good predictors for which undergrad country) are consistent with what students who studied in those countries have told us about grading standards, student culture, and exam prep. So, we really have no reason to doubt them.
share|improve this question
2  
@scaaahu We are not - the school is also considered in the analysis, but turns out to be less (statistically) important for our particular applicant pool. –  ff524 Mar 3 at 4:53
5  
Of course it is. Moreover, it's unfair to apply the same criteria to all students from the aame country, instead of varying the criteria by the undergrad institution. Alternatively: You aren't using different criteria; the criterion for all applicants is their predicted GPA. –  JeffE Mar 3 at 4:55
1  
@JeffE every answer so far disagrees with your comment - care to elaborate in an answer? –  ff524 Mar 3 at 10:19
5  
@Populus: It's well-known that GRE and other "standardized" tests are not administered in a standardized way worldwide. And why would it be up to a single department recruiting students to try to fix the irregularities in the GRE? –  Ben Voigt Mar 3 at 15:16
2  
@Populus We're not proposing to use geography as a criteria for admission; we're suggesting that different criteria measure skill and knowledge in different parts of the world, and wondering how we can fairly account for that. –  ff524 Mar 3 at 15:33

8 Answers 8

I'm going to advise against.

  • You say your sample size is small. And honestly, unless you hired a professional statistician, I have doubt the analysis was carried out properly or shows exactly what you think it shows. Good statistics are hard to do.
  • Assuming my google results are truthful, National Origin is a Protected Class in the US. You can't discriminate against people because of it. I assume this is an american institution as you talk about GRE, but similar applies in other places. EDIT: OP has since clarified a misunderstanding of mine - the judgement is based not on the student's country of origin, but the country of the school they attended, which makes me less certain this point still applies.
  • You're department is actually, seriously proposing to say to people "Sorry, while you have the same official qualifications as another candidate we accepted, we're rejecting you because of your country of origin"? I mean, really? Whoa. Just stop and play back how this kind of justification would sound in almost any other circumstance. Think about how the press will make it sound when they (inevitably) get wind of it.

SUBSEQUENT EDIT

So, this question interested me, and I've bounced it off a couple of other people in my lab. Their main concern seemed to be that grouping by country was just to coarse a measurement, as every country has its own mix of good and bad schools. The basic idea, though, didn't seem to elicit the same gut reaction as it did from me.

Moreover, I've googled and found at least two examples of courses that vary their requirements based on country of origin, so there is precedent:

https://www.auckland.ac.nz/en/for/international-students/is-entry-requirements/is-minimum-overseas-entry-requirements.html

http://sydney.edu.au/business/futurestudents/postgraduate_study/pg_coursework_studies#app_req

So maybe I've got this wrong, and should just be ignored. At any rate I'm no longer sure it's quite so clear cut.

share|improve this answer
    
"Students who earned their undergrad degree from a university in X" Okay, that's a better criteria than what I thought you were doing at first (The line "Is it fair to apply different criteria to students from different countries in admissions decisions?" really confused me on the front, as there's no mention of institution there - only the student). However, having a blanket rule on universities based on country they're in still seems to have massive potential to blow up in your face later to me, legally or otherwise. –  Pat Mar 3 at 9:33
3  
Yeah, it's not I problem I envy you for having to deal with. What kind of numbers of students are we talking about processing here? Tens? Hundreds? Thousands? Would it be possible to set a departmental test they all have to carry out and judge them on that? Or run interviews? –  Pat Mar 3 at 9:46
1  
@shane it's really not that simple. For example, how would we verify that students submit their own work? Who says ability to solve one particular programming problem predicts success in our M.S. program? (I don't think it would). –  ff524 Mar 3 at 14:45
1  
I think the last point would be more like: "So, you got the same score as this other candidate, but as we think you have had to overcome greater obstacles, and your university has prepared you better for this job than the exam, we are taking you in instead". –  Davidmh Mar 3 at 19:16
1  
That is, it is invalid (statistically) to argue about discriminatory impact by pretending to hold everything but the national origin constant, changing the national origin, and observing that this would change the applicant rating computed using the model. If the variables include GRE Verbal or TOEFL, it is essentially impossible to change an applicant's nation from China to Canada without altering those scores. The whole point of the statistical model is to account for the difference in the interpretation of such scores as detected by performance measures that are nationality-blind. –  zyx Mar 17 at 7:31

I don't think it has to be bad. GRE is a measurement of how good you are at doing exams, and not necessarily the most relevant kind to your graduate work. Some universities and education systems are very good at making people excel at set fixed format exams; but they don't teach them how to think by themselves and be creative.

In my opinion, GRE is a very biased measurement in itself, and it is not very well designed (just look at the ridiculous maths part). Also, it is based on a lot of fairly simple exercises, whereas in some universities, we are used to completely different kind of exam: a few long and complicated exercises, and perhaps don't have the ability to work fast enough in simple and repetitive tasks.

share|improve this answer
2  
I hope by "Ridiculous maths" you meant "ridiculously easy, to the point where it fails to differentiate great applicants to technical programs from mediocre ones." Non-technical programs oughtn't to be using the math score for much of anything. –  Ben Voigt Mar 3 at 15:22
    
Yes, that is what I meant. I haven't look much at GREs, but judging from the maths one, I fear many parts will likely be useless measurements: or too easy, or too specific for a certain background. –  Davidmh Mar 3 at 18:36
    
If the GRE is too easy you need to find another test. This is not an argument for biasing it against people from certain countries –  Philip Gibbs Mar 3 at 18:52
4  
It is not only how difficult it is, it is also what it measures. Changing GRE for something more sensible would be the ideal option, but it is way outside the scope of a department. The question here is if they could, with what is available, make a better choice. –  Davidmh Mar 3 at 18:56
1  
@PhilipGibbs, the purpose and quite possibly the result of the statistical analysis is to un-bias the GRE (which favors or disfavors various nationalities, languages, types of education system, etc) by validating it against real outcomes that are defined uniformly with respect to nationality, such as performance in the program. It is absurd to imagine that national origin, either per se or using undergraduate institution as (in effect) a proxy, does not play a huge role in the way that, among many other things, GRE verbal scores should be weighed. –  zyx Mar 17 at 7:37

I was waiting and hoping that @JeffE would expand his comment, since I share the opinion he expressed there, and moreover I think he is far more qualified to give advice on the subject. (Quite possibly, if he decides to expand his comment to an answer that has better facts and arguments than what I am about to offer, I will delete my answer).

DISCLAIMER: I do not know either where you are, what the laws are like there, or if your final decision can have any legal repercussions. I advise seeking legal counsel for any questions of that sort.

I think it if fine to have different criteria for students that obtained their undergrads in different countries (and/or in different institutions), as long as you can ensure the criteria are fair.

Now, how to approach ensuring that the criteria are fair, I have no idea, but I definitely know that if in some countries the GPA is indicative of a person who has potential as a researcher, in some countries it is simply not (I for example did my Masters in Croatia, and I know both people with around 3/5 GPA that made excellent PhD candidates and people with a perfect 5/5 GPA that I do not think would be capable of much independent thinking that were happy to take jobs where they have strictly defined output they have to produce and no research to speak of).

I have two different examples of using "different" criteria in the admission process, but unfortunately both of them are for the wrong level of study:

  • First one is about undergrad admissions to my former Computer Science Faculty* in Croatia. While the big majority of the candidates take a standardized test and are admitted based on that, a small amount of students (maybe 1%, maybe even much less) are invited.

    These invitations work as such: the Faculty regularly does a continuous performance study on the previous invitees according to their High schools, as well as (I assume), overall performance of students from different High schools. Each year, a number of invites is extended to High schools, who can then "award" their students with those invites.

    Based on the performance study and possibly High school size, some schools get a larger number of invites (up to 5, I think), while some get only 1 invite or even none.

    And, actually, I think it's fair. E.g. the strong mathematical High schools will get the most invites (those are the schools "prepping" their students for technical studies after all), strong general schools would get some, and weak schools would get none. And still everything would be re-evaluated year after year. Also, additionally, nobody still loses their fair chance to enroll: this "invited" students make less than 1% of the enrolled students, while everybody can still take the standardized test.

  • The second example is about the interview for a PhD position / pre-PhD internship.

    Recently, a permanent professor from my lab started looking for a person to hire for an internship with a strong chance of offering them a PhD upon the completion of the internship. I mentioned that my ex supervisor used to supervise people with a similar profile to what was required, and that I could ask him weather he has somebody interested (and good enough) to apply.

    When he received the preliminary application documentation, he asked me to comment on the profile because he was not familiar with the Croatian University system at all. I said that good grades were usually indication of a good student, but bad grades do not have to be an indication of a bad student.

    There was also some other points in the CV adding value to his application, not directly obvious to somebody non-Croatian. After talking to an ex post-doc of his (who worked a bit with the applicant), the professor decided to interview with the student.

    A few days later, he told me that based on the interview, he offered the applicant a position. He also told me that he would most probably not interview a person with that profile, if not for what me and his Croatian ex-post doc said about the "interpretation" of the profile.

I agree with both these cases. The ("absolute") criteria is indeed not the same for everybody. But still, in both these cases, the goal was to be fair, to base the decision on the applicants abilities, and finally to judge the applicants abilities and potential "on the same scale" for all applicants, just based on different information that was available, finally causing different "absolute" criteria.

*In Croatia, the Universities are not wholesome entities, and all the administrative decisions are made on the level of the Faculty. There is no identification of students with the Uni in Croatia; if you ask a Croatian studying "at home" where he is studying, he is going to provide the information about his Faculty.


ADDITION. I wanted to add that, despite what any statistical research might show, if students from different countries have the same, internationally standardized tests, then I think making a different criterion would be wrong.

Maybe, for the country X, those tests are not good indicator because students get prepped exactly for those types of exams, but there just might be a few students who learned the material in the "proper way" (as do the majority of students from the country Y), who might get rejected for no real reason.

So, bottom line, I think

  • studying, and then declaring your own criteria to interpret different (non-standardized) national grading schemes is fine (even if they use the same scale "on the outside" -- e.g. two countries which both have a national standard grading schemes on a scale 1 to 5)
  • but, if a criterion is based on an internationally standardized exam, that same criterion should be used with everybody where applicable (e.g. everybody who took that standardized exam).

    As a compromise-suggestion, I think it would be fine to say that you accept all students from everywhere if their score is extremely high (e.g. >95%), and all the students with medium-to-high scores (e.g. between 75% and 95%) will be considered based on additional criteria (where you can introduce personal interviews, standardized test from your institution, personal research statement + recommendation letters)...

Edit2: This opinion was supported when I talked to a person with a background in law (not in academia thou).:
The final verdict was: interpreting criteria that are initially different (e.g. GPA) differently is OK, but if something is standardized on an international level, it would only be OK to either take it in to account equally for everybody, or not at all. (purposefully using OK instead of "allowed" or "lawful" since I don't know the laws at your place). With a note that, unfortunately, What is lawful and what is just and fair is sadly not always the same.

share|improve this answer
    
The specific example you mention, of asking a trusted individual from country X for advice in evaluating students from X, is I believe very common. I think what we're trying to do is essentially the same thing, though less subjective. –  ff524 Mar 3 at 18:45
2  
Exactly. The point was finding the "context" in which to evaluate the student. The same way the first (undergrad) study is trying to put the schools in context (i.e. you might get an invited admission if you're an excellent student from an excellent school, but not from an average school), the second one is trying to put the transcript in context (i.e. does the mediocre transcript by absolute value really mean a mediocre student or not). And of course, in your case of a large student pool, such a context has to be based on a large pool of "opinions" - evidence, and not on one individual. –  penelope Mar 3 at 18:58
    
Neither of the examples actually answer the question on whether or not you can officially demand applicants to meet different criteria based on where they come from. More specifically, #1 would be considered unacceptable and immoral in Scandinavia, it would be some serious sh*tstorm if something like that were to come up on the news. #2 happens on a regular basis pretty much lab I know of. It does not mean that the professor in question can go out and write it out in plain text that if the applicant is, from China/India/Croatia/Madagascar/... they don't need to meet the same criteria. –  posdef Mar 3 at 22:52
    
Consider asking the professor in question whether or not he would or could write the job advert in the following way: "The applicant should have a GPA of 3.5/4 (~87.5%), unless they are from Croatia, in which case 3/5 (60%) is good enough." –  posdef Mar 3 at 22:54
    
Well, actually, the 3/5 GPA in Croatia is more like 75% since the grade 1 is a failing grade... But again, not really, since all the passing grades are distributed as in lowest 15% of the students gets a 2, next 35% gets a 3, next 35% gets a 4, and the best 15% a 5. So, "yearly score" of 3/5 actually means that the student is "somewhere between the the best 75% and 50% of best students who got all passing grades"...or something similar like that. I'm just trying to demonstrate that to recalculate the GPA of ~87.5% to Croatian system is really not so trivial and obvious. –  penelope Mar 4 at 12:59

I think the underlying questions are (ought to be) first, whether it is fair to reject a student because you cannot accurately measure their potential to do well in the program; and second, if you can perform equally accurate measurements on two groups as long as you use different criteria, is it okay to do so?

Answering the second question first: I think the answer is "yes". The reason is made apparent by advocating the alternate view: should you take students who are likely to do poorly simply because you fail to apply a more sophisticated measurement? That seems pretty boneheaded, and not very fair to the best students. You should leave the analysis up to computers, as they are good at this sort of thing and impartial, but it's a good idea if you can implement it. Don't forget, though, that data on whether a score distinguishes students within your program does not tell you about whether it helped you reject students that didn't make it in! So you should be skeptical but open to the idea if that's where the data leads.

The first question could be rephrased as: "if I know I'm not getting a good sense of some students, can I just ignore them all?". And that guides my advice there: no, that is not fair. Find a way to do your job better--to get more information so that you can get a good sense of these students, or just accept that you would rather make mistakes in acceptance than find a way to do better on your predictions.

Putting these two together: if you have equal statistical power across groups when subdividing one population, then great! Use the information. But if you end up with one group better-measured than the other, you should only apply different criteria if you can show that students who are great and do all the right things in their less-measured context still have a shot at being admitted.

(Also, simple thresholds for individual scores, e.g. "We only take people with GREs above X" are rarely a smart way to run admissions unless those thresholds are set amazingly low. You should be thresholding an overall score to get the top N students, so you'd be applying different weights for GRE in one context vs. another.)

share|improve this answer
    
simple thresholds for individual scores - yes, I simplified for purposes of the question - the classifier is more sophisticated than that. –  ff524 Mar 3 at 16:08

You asked an ethical question and got a lot of scientific, legal, and political answers.

Ethically, you should:

  1. Do your best to admit people based on merit.

  2. Be open about your policies.

  3. Make a serious and competent effort to ensure that your statistical methods are valid logically and empirically, and that they can withstand scrutiny from people who are experts in the field.

  4. Carefully consider the historical legacy of racism, colonialism, and nationalism, and work hard to make sure that you aren't inadvertently reinforcing this legacy.

Your behavior so far is probably far more ethical than that of most people in your position, since most such people are probably secretive about their practices (you publicly asked for advice) and probably apply various heuristics without carefully considering whether those heuristics could withstand professional scrutiny.

You might want to expand this small, informal, unpublished, nonprofessional study into something more serious and systematic, done by people who have expertise in psychometrics and the (very difficult) methodological issues of the social sciences.

You are unfortunately operating within real-world constraints imposed by (1) the existence of countries where GRE scores are fraudulent, and (2) the existence of countries with such poor undergraduate education that undergraduate GPAs don't mean much.

share|improve this answer
    
I publicly asked for advice - under an anonymous username. Not sure how much that's worth :) –  ff524 Mar 31 at 3:28
1  
@ff524 I don't know what others think. It's worth a lot to me regardless of the anonymous username. At least I know what some other opinions are. –  scaaahu Mar 31 at 3:37

I would argue that you are interpreting your data incorrectly and using the results in an unethical and discriminatory way. Your model is not identifying groups of individuals who should not be accepted, but rather groups of individuals that require additional support so that they can reach the full potential predicted by their past performance.

Consider the following example: For a variety of reasons that have nothing to do with ability or commitment, women have historically been less likely to become full professors in STEM fields than men. To reject female applicants base on a model that captures this historical effect is completely unethical. What the model would show is that female applicants need additional support to maximize their potential (e.g., flexible working hours and mentoring).

share|improve this answer
2  
I don't think it shows that at all - it's more like saying, "GPA doesn't historically capture the ability of female applicants to succeed, so we should consider another criteria, like work experience, in evaluating them." –  ff524 Mar 3 at 16:41
3  
I would argue that you do not know which data the OP has, how he has done the analysis and on what exactly is his interpretation based. Try to, just for a second, imagine that his interpretation is correct. If you apply your reasoning to e.g. sports, would you rather compose an Olympic team from 5 best sportsman in the country that accidentally all practice in the capital, or would you say that this results are skewed and entice favoritism towards capital-dwellers, and instead rather train 5 sportsman who were only good in their local city/county, in order to "reach their full potential"? –  penelope Mar 3 at 16:47
    
@penelope I would want to train the 5 best sportsman independently from where they were from. I would not care that historically the only people who win Olympic medals are capital dwellers. If the 5 fastest and strongest (or highest GPA and GRE) happen to live in the capital so be it. –  StrongBad Mar 3 at 16:54
2  
That's not what the data shows - the data shows that the value of every predictor we have available varies (statistically significantly) according to the country a student previously studied in (at least, for our applicant pool). Answers that assume this isn't possible aren't really helpful to me, they don't address the question. –  ff524 Mar 3 at 17:02
2  
@StrongBad As far as I can see, the OP made a study of predictors. And he is saying that the best predictor is not GPA, but something far more complex, and the country of undergrad study plays a significant part in that predictor. As far as I understand, we are not here to try and tell him that he did a wrong study (because some of us might not like the results), but rather to tell him what to do with the study results assuming they are correct. If he wanted to asses the validity of his study, I'm hoping he would have gone to a different SE site. –  penelope Mar 3 at 17:14

Does the answer change if this would significantly skew the admissions decisions in favor of a particular country of undergrad study (because statistically, applicants to our program whose undergrad degrees are from X have done much better than those with undergrad degrees from Y).

My concern is that we're effectively saying, "Students with undergrad degrees from Y with a GRE score < T will be rejected, but students with undergrad degrees from X with GRE < T may still be considered for admission (pending other criteria)."

To my ears this sounds analogous (with certain exaggeration) to: "we only want rich, white men from successful parents that could afford to send them to good schools through out their childhoods."

I realise that this is not what you intend but you got to realise that having different criteria based on the country of undergrad studies is unethical, biased and I can't see how you could implement something like that without having major headaches. You can, by all means, have different criteria for an individual (for instance by having an interview) but if you clump up an entire population based on some statistics, which you can't elaborate on for some reason, I call that discrimination.

Bottom line is, you cannot evaluate the chances of an individual being successful or not, without giving that person a fair chance. If you deem a particular GPA/GRE score to be "good enough" for your graduate program, all applicants that satisfy that criteria should be considered good enough, regardless where they come from. Any additional selection criteria could be justified only if it warrants additional information, such as TOEFL score for those who don't have English as a mother tongue.

One other option would be to calculate "success rate coefficients", something like a multiplicative factor for the GPA/GRE score for applicants from different undergrad institutions, which could in theory be a fair assessment, but practically unfeasible considering the number of possible institutions involved.

Another alternative would be to devise a test for your institution, that you consider to be more fair than using GPA or GRE score. But even that, judging by your comments, is not an acceptable solution. Honestly it sounds a bit like you just want confirmation of some sort.

share|improve this answer
4  
It's "Country of undergraduate study," not the applicant's nationality. How do you propose we adjust undergrad GPA for different grading standards between universities if we apply a single numerical criteria to all? –  ff524 Mar 3 at 9:21
1  
The point is we don't have any single factor we can rely on for everyone - we've found that the indicators of student success are different for different populations. Hence, the question. –  ff524 Mar 3 at 9:26
1  
I understand the essence of the question, and the need to ask it, I do... and my answer is that if you are operating within the boundaries of fair treatment and equal chances (which you should) then it's not as simple to change the thresholds and change the criteria depending on the candidate, because it is not fair to the person in question. If it is about evaluation of significance of the GPAs, you can do an assessment of several hundred universities from which most of your applicants come from. But it is a daunting task –  posdef Mar 3 at 9:46
3  
Huh? You think the meaningless numbers on the GRE are absolute indicators of research potential? I've never heard of a good physics program in the US even looking at GRE's from, say, China, because experience has taught that every student there memorizes the answers, whether the student is capable of doing research or not. So as long as people game the system along national lines, it's discriminatory to control for it statistically? –  Chris White Mar 3 at 20:45
1  
we only want rich, white men Please don't make inaccurate, offensive characterizations of what the OP is trying to do. Please don't impute racism to the OP when race was never mentioned. Please don't impute classism to the OP when class was never mentioned. Please don't impute sexism to the OP when sex was never mentioned -- and BTW sex is not strongly correlated with the country in which one does one's undergraduate studies. –  Ben Crowell Apr 1 at 5:39

I would "massage" this problem by including a dummy statistical variable for the country. E.g. 0 for the country where grades are more important, and 1 for the country where GREs are more important. You could almost separate the applications into two piles this way, and tackle one pile at a time.

If asked about it, I would answer that neither grades nor GREs alone are dispositive, and they both have have greater explanatory power when you introduce the third (country) variable into the equation.

As far as I'm concerned, it is ethical to use any combination of statistical variables that satisfactorily explain performance, and to strive for "best fit" (statistically).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.