14

The Internet is littered with information about university rankings and comparisons that it is very easy for an applicant to lose track of what exactly he wants to know. Apparently rankings which compare all departments of a university and have a single rank are not very useful from an applicant's perspective. Yet these are the most common rankings one can find. I know of the FT rankings which order MBA schools. It does rank B-schools in terms of research too, but clumps different branches (Accounting, Finance, Operations, etc) into one.

  1. For other departments are there reputable and reliable portals where one can get to know rankings based on various criteria like published research, number of graduates, time taken for graduation, etc?

  2. Are there no well-established ranking systems for academic departments? For example, these ranking systems could be similar to those in cricket or football. Such as periodic updates are made with every publication and every citation, and possibly journal reputation could also be brought into picture. In a way, would that not enhance the competitiveness in academia among similar departments?

21

The big difficulty with devising formal ranking systems based on numerical measures is that, outside of a handful of areas like sports, anything we can measure is at best a proxy for what we really care about. It may start off as a pretty accurate reflection, but anyone judged on this basis will quickly discover how to manipulate it.

For example, universities in the US are often judged partially by the fraction of students they accept. Of course, ambitious universities have adapted by advertising to encourage more applications, with no intention of accepting these applicants, but just to lower the acceptance rate by increasing the denominator.

Similarly, universities are also judged based on "yield," the fraction of admitted students who attend. That sounds at first like a pretty good measure of popularity, but it creates an incentive to game the system by rejecting students you think are likely to choose another university in the end, and universities do just that.

Time taken to graduation can be gamed by kicking out students who are taking too long. Employment rates can be gamed by offering ten-week temporary jobs to unemployed students, timed to coincide with the measurement of employment rate (really!). All sorts of things can be gamed.

And this is not just a theoretical problem. It occurs all the time in practice (if you are in the US, then your school is very likely doing some of these things), and some people are extremely upset about it.

You might think scholarly measures based on citation counts would be less subject to this, but they are not. There are plenty of corrupt journals where editors put pressure on people to cite papers, or even use totally fraudulent methods like publishing review articles that carefully cite every paper they have recently published. People have been caught seriously distorting their journals' impact factors by doing this, and I'm sure there are other, more clever editors who are getting away with it. If professional success depends on influencing a number, then people will discover ways to influence it.

In summary, there's an awful tension between transparency and resistance to fraud: if you explain how your ratings work, then people will manipulate them. Nobody has any idea how to avoid this, and the net effect is that serious scholars do not waste time trying to compile numerical rankings. For the most part, the only people who do are those who are naive, trying to make money in unscrupulous ways, or trying to promote a cause through carefully chosen rating methods. The rankings they produce are not worth paying attention to.

P.S. Polls of expert opinion are generally much better than rankings based on numerical measures, but even they have their problems. For example, the U.S. News college rankings are based partly on asking college presidents to rank other schools. Clemson University manipulated the rankings by rating all competing schools as below average, no matter how good they were. I suspect they weren't the only ones to do this - the amazing thing isn't that it happened, but rather that we ever found out.

  • Wow - I never knew things are this bad, w.r.t inflation of rankings! – TCSGrad Apr 21 '12 at 4:24
  • @Anonymous Mathematician: Excellent answer, that was highly useful :) – Bravo Apr 21 '12 at 6:06
  • 3
  • 10
    By the way, I forgot one of my favorite examples of crazy incentives in ranking, from freedom-to-tinker.com/blog/felten/newsweek-fails-ap-math. Newsweek has ranked US high schools using the number of AP or IB exams taken divided by the number of graduating seniors. Under this system, every time someone shows up for a exam they are unprepared for, the school's score goes up, but every time a struggling student makes it to graduation rather than dropping out, the school's score goes down. – Anonymous Mathematician Apr 21 '12 at 21:05
  • What is the rationale for including a University's popularity among high school graduates in the ranking criteria? – Jase Dec 22 '12 at 13:36
7

To answer your second question, getting accurate data can be tricky. The recent kerfuffle over the NRC rankings of computer science is a good example. The methodology was flawed, the data was old, and the data was misinterpreted, and it still took many years to produce the results !

  • 2
    ...the data was badly sampled... – JeffE Apr 21 '12 at 19:33
  • You should link to and discuss the NRC more fully. Yes, it's problematic but it warrants further explication. – RoboKaren Aug 4 '15 at 15:13
2

I can add your list with two more well-recognized ranks:

They indeed present some divergent results, but I guess they may provide community with some clue about the "best" universities in the world.

Indeed, I agree with you that it'd be more interesting not only check out these "marketing" ranking, that considers a bunch of aspects, sometimes not transparent (!). IMHO, a good "ranking" would depend upon your subject matter, i.e., the research of interest.

You really posed a very nice question, and I'd appreciate to hear from colleagues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.