What is a common way to tell if an academic is "above average," or outstanding in their field? Is their h-index (Hirsh index) one of them?

Question

If one claimed that a particular scholar was "above average" or "noted" in their field, is there any good metric by which to support or deny such a claim?

Is there a generally accepted way to indicate that a particular professor or scholar is outstanding, or above average, in their field? I understand there are certain indicators, such as chairs, endowments, prizes, etc. But these don't really seem to help to compare one scholar to another, except in a sort of gross, simple count way (i.e. one professor has had more chairs than another)

Is it theoretically possible to create a "ranking" of professors in their fields, by some metric? Could their h-index serve as such a metric?

Answer 1

Yes, there is one and only one standard method that is universally employed by reputable academic institutions worldwide. This is how you evaluate a researcher:

1. Read their papers.
2. Attend one of their talks.
3. Ask the opinion of other experts in the field.

This is how hiring committees and promotion committees do their job. There are no shortcuts. Parts 1-2 require that you have some relevant expertise; if not then you must rely entirely on #3.

Every academic is regularly asked to give an expert opinion through reference letters, which often are expected to include some sort of ranking (e.g. "Assistant Prof. X should be promoted because she is clearly as talented/better than Prof. Y who was recently promoted at Prestigious University Z"). How does one justify this kind of claim in the reference letter? You guessed it:

1. Read their papers.
2. Attend their talks.

Answer 2

If one claimed that a particular scholar was "above average" or "noted" in their field, is there any good metric by which to support or deny such a claim?

No. As a rule of thumb, this isn't the kind of thing that you can measure with a metric. Elvis Presley was the king of rock and roll. Why? Is it because he pumped out more albums than the others? Because he sold more? Because journalists wrote more about his albums than the others'? No. It was because he was the king and few people contested that. It's the same in academia. Either you can say that someone is "noted" in the field and be reasonably confident that you won't be contested when saying that, or you can't. If you can't, then you should avoid it, on pain of looking pretentious or like a toady.

Is there a generally accepted way to indicate that a particular professor or scholar is outstanding, or above average, in their field? I understand there are certain indicators, such as chairs, endowments, prizes, etc. But these don't really seem to help to compare one scholar to another, except in a sort of gross, simple count way (i.e. one professor has had more chairs than another)

In general, you can't compare people. This isn't a video game, people don't have a numeric level associated to their academic ability, someone with a 12 being better than someone with a 5. It doesn't work like that. There is a multitude of factors, most often not measurable or not comparable. Trying to make a sum out of these and comparing the result for two different people will only lead to crap. Ask any hiring committee if determining who is the best candidate for a job is easy, let alone determining who is the best researcher.

Is it theoretically possible to create a "ranking" of professors in their fields, by some metric? Could their h-index serve as such a metric?

God no. Of all the metrics, you've probably picked one of the worse ones. If I write two dozens pieces of trash that all cite one another and publish them in vanity press, I will have a great h-index. Will I be a good researcher? No. On the other hand, if I write a single article in my whole life solving the Riemann hypothesis, then I would probably become one of the most famous mathematician in the world overnight, but my h-index will be crap.

Answer 3

If you want to rank two professors against each other, you might be tempted to use the h-index. Don't. As many of the other answers point out, it's a severely flawed metric, and it doesn't really tell you a lot.

However, if you want to figure out whether a given professor can reasonably be described as "noted" or "outstanding", then that is a quite different question. And here, yes indeed, I would say that you can use certain indicators, namely awards, honors and prizes. I do not think anyone disputes that a scientist holding a Nobel prize is outstanding. (Peace and literature, maybe not so much.) If a mathematician wins the Fields medal or the Abel prize, the same.

Many societies award fellowships. To get one of those, you have to demonstrate academic excellence, and often also things like service to the society in question, outreach, teaching etc. The advantage is that the "overall package" a professor offers has already been evaluated by people who are presumably experts in the field. For instance, here is a list of the Fellows of the International Institute of Forecasters, which I happen to be involved with. Some of the Fellows are a bit contentious, but nobody from the field would dispute their being noted.

Best paper awards are similar.

Of course, you need to use a little expertise in deciding whether a Best Paper Award from a journal on Beall's list is truly a mark of excellence, or whether a Fellowship from an academic society that offers little more than a one-page webpresence is. But unless you go with the extremely well-known marks of excellence like the prizes I noted above, there is simply no shortcut that will avoid having at least a passing knowledge of the field.

And note that this allows you to decide whether someone is distinguished or not. It won't tell you whether A is "more distinguished" than B, like one might try to use the h-index to indicate. Which, as I argue above, is impossible.

Answer 4

Is there a generally accepted way to indicate that a particular professor or scholar is outstanding, or above average, in their field?

The generally accepted method for assessing a particular scholar's merit is to familiarize oneself with their work. Such an assessment requires a solid basis of expert knowledge.

I understand there are certain indicators, such as chairs, endowments, prizes, etc. But these don't really seem to help to compare one scholar to another, except in a sort of gross, simple count way.

This applies to any "metric", although some are worse than others. Any sound assessment would have to be qualitative and require some substantive engagement with the scholar's work. Therefore, any comparison, to the extent that it would be useful at all, could only point out qualitative differences that don't lend themselves to a ranking, "except in a sort of gross" way.

Answer 5

The negative proof to the question here is far broader than academics: is there a metric for the best car? Best parent? Best programming language? Smartest person? No, because all these things have many orthogonal dimensions that simply can't be collapsed to one without unacceptable information loss. Researchers can be creative, well funded, methodical, hard working, well versed in literature, collaborative with peers/students, etc.

I concur with David Ketcheson's answer on what to do instead.

Answer 6

All metrics that used (e.g. number of first/senior authorships, sum of impact factors, percentile ranks of impact factors, citations, H-index, grants and other funding etc) have all their advantages and many more disadvantages. Never the less they are used in hiring processes in one or the other way because otherwise it is not possible to assess several hundred candidates that apply for a faculty position. Which of these factors are important in a certain sub-field is very different. Only the ones scoring top in these metrics will make it to the interview where then other factors might count as well.

For someone who is not familiar with a certain field the easiest (but still not always correct) way to see how good a professor might be is the name of the university. e.g. a professor at Cambridge will most likely have achieved a lot in his life. Someone at a no-name place will not have made much impact that impressed other people in the same field and if such a person does make a big impact one day then he will most likely get offers to move to a place with a better name.

Answer 7

If one claimed that a particular scholar was "above average" or "noted" in their field, is there any good metric by which to support or deny such a claim?

The only "generally approved" quantitative metric is the h-index. H-index is a metric, is OKfor your task as it allows you to define above or below average. As a matter of fact, this is the way some national educations systems stamp their professors as good enough for tenure. It is also agreed that it is not "good enough" - famously, Peter Higgs, 2013 Nobel in Physics, would fail miserably a ranking based on h-index only, as he published very few paper, although with huge citation count. Also, h-index is a measure of lifetime achievements, thus needs to be corrected for the academic age. Which brings us to the next point.

Is there a generally accepted way to indicate that a particular professor or scholar is outstanding, or above average, in their field? I understand there are certain indicators, such as chairs, endowments, prizes, etc. But these don't really seem to help to compare one scholar to another, except in a sort of gross, simple count way (i.e. one professor has had more chairs than another)

Other, mostly qualitative metrics are regularly used, consciously or not, in academic's minds, although no official ranking exist. I will mention a few, the ordering only reflecting the stage in an academic career:

1. Institution where PhD has been obtained
2. PhD supervisor
3. national prizes 
4. national grants
5. number of PhD students supervised
6. chairs at institutions or conferences
6. international prizes
7. academic success of PhD students mentored
8. more I could not think about now :)

Is it theoretically possible to create a "ranking" of professors in their fields, by some metric?

Of course it is, there is entire field about it called Scientometrics. You have to 1) fix for h-index known limitations 2) combine with the variables above to come up with a more comprehensive algorithm that will rank any researcher in any field. The reasons why this has not been done before are twofold. First, it is not easy at all to define objectively how much every metric listed here should weight in the ranking algorithm. Second, and most importantly, academics rank every day for jobs, promotions, accepting papers or conference contributions, prizes etc. However, they prefer their ranking algorithm to suit their individual minds, rather than adopting a common framework.

Could their h-index serve as such a metric?

As described above, h-index has many limitations that make it impractical for most purposes. But an entire field of research exists around it - Scientometrics - so rest assured there will be developments.