What is a correct way of evaluating the effectiveness of criteria for admissions?

Most colleges and universities in the US base their admissions on high school grades plus a standardized test (SAT or ACT). To decide whether a particular piece of data X is a good admissions criterion, people seem to focus on how well X correlates with college grades. (This may be freshman grades or total college GPA. Some people think success in getting a degree would be more appropriate. For the purposes of this question, I don't think these distinctions are relevant.)

Isn't this correlation a completely incorrect way of evaluating X as a tool for admissions?

It seems like an apples-to-oranges comparison. A student who has low X is admitted to a nonselective school, where standards are low and they compete with other low-X students. Grades at this school measure how well they did based on the low standards prevailing there. Students who have high X are admitted to a selective school. A "B" grade at Berkeley is not the same as a "B" grade at Cal State Dominguez Hills. A high-X student goes to Berkeley and gets a "B" in calculus. A low-X student goes to CSDH and gets a "B" in calculus. This shows up in the statistics as a lack of correlation between X and grades, since the students differed in X but got the same grade. But isn't that misleading, since the grades mean different things at the two schools?

Is there some more logically justifiable statistic to use in evaluating whether X is a good admissions criterion?

• Bingo. This certainly identifies a nonsensical aspect of "higher education". I have no simple solution... but can confirm that the nonsense is real. So whatever decisions we/anyone makes, if we want them to be genuinely sensible, should take this nonsense into account. (Usually this is "impossible", a.k.a. impossible to document un-prejudicially...) Commented Mar 31, 2014 at 2:20
• You could measure applicants against how well students matching a similar multi dimensional profile do at your school but that can be very controversial, apparently Commented Mar 31, 2014 at 2:29
• Since most schools are ranked, couldn't one create: `score = f(grade, school rank)` and see how `X` affects `score`? Commented Mar 31, 2014 at 3:38
• Basic rules of statistics are violated: there is no control group. You'd need a school that manages to admit everybody and provide everybody with high-quality teaching (as opposed to, let's do one year of admitting everybody with our current staff situation so we can figure out good criteria), then correlations would be meaningful. Commented Mar 31, 2014 at 12:10
• @Raphael: You'd need a school that manages to admit everybody and provide everybody with high-quality teaching This sounds like a description of a community college. But regardless of the quality of the teaching, community colleges have lower standards.
– user1482
Commented Apr 1, 2014 at 5:34