Do schools/departments ever evaluate their own admissions process?

Question

I'm curious as to whether or not any departments ever attempt to evaluate their own admissions decisions. It seems quite plausible to do for any college or graduate program, but I've never heard of it done. I have in mind some kind of data analysis comparing student success in the program or beyond with variables from the application.

I understand some might not like the idea, but as noisy as admissions are, I was wondering if schools ever reflect on their own decisions and attempt to identify strengths and weaknesses?

Answer 1

This practice is not frequent, and it is rarely systematic and rigorous. However, from a decision-making perspective, this has been given attention by Robyn Dawes, who was a very well-known psychologist at Carnegia Mellon. Here is one of his papers on the topic, which has been cited about 500 times (per Google Scholar). Some recent evaluations have been published, but relative to the total number of admissions committees that exist across all graduate schools, I can confidently say it is not many.

Dawes, R. M. (1971). A case study of graduate admissions: Application of three principles of human decision making. American Psychologist, 26(2), 180.

Answer 2

To me the best evaluation of an admissions process would be the success of students once through the system. I am not sure what would otherwise be evaluated? Perhaps that can be expanded upon in the question? In any case, I am sure many departments run tabs on where alumni go and I can only share the statistics from two departments my Alma Mater and my current workplace.

Figure 1. Statistics on post-graduate careers of students from the Geology department of University of Minnesota

Figure 2. Statistics on post-graduate careers of PhD students from the Physical Geography and Quaternary Geology department at Stockholm University

So from this perspective the admissions system is evaluated from the perspective of employability within the field.

To study the process itself one would need to set up some additional criteria against which to evaluate the process but I am not sure what those criteria would necessarily be. Some aspects such as gender and ethnicity bias are of course identified but it is unclear if there is a need to try to actively change the interest of applicants other than strive to reach under-represented groups. Such outreach activities are, however, not part of the admission process.