6

I am wondering if there have been any sort of rigorous/meaningful study about how well an h-index measures the scientific productivity of a person. I am wondering how much stock I should put into the h-index, both as a personal measure and a measure of those around me.

  • 18
    Having the ability to quantify how well the h-index measures scientific productivity would by proxy imply that there exists some reliable measure of scientific productivity to use as a control. Unfortunately, no such reliable measure exists. – Moriarty Mar 16 '15 at 15:13
  • 5
    Is the h-index intended to indicate productivity at all? I think a researcher can be tremendously productive (in terms of publication frequency, number of findings, etc.) and yet receive very few citations for one reason or another. – O. R. Mapper Mar 16 '15 at 15:16
  • Relevant ( see the answers for why not to do this) : academia.stackexchange.com/questions/5687/… – seteropere Mar 16 '15 at 18:31
  • Why on earth would you want to measure scientific productivity anyway, as opposed to scientific impact, or better yet, scientific insight? – JeffE Mar 17 '15 at 21:36
  • @JeffE, depending on how you define scientific productivity it could include those aspects as well. At least that's what I've been thinking, maybe I'm wrong and that's what's producing some of the confusion here. – NeutronStar Mar 19 '15 at 15:31
5

Besides the convincing argument in Moriarty's comment to the OP there is also a formal argument that the index is particularly meaningless. The Notices of the AMS recently published "Critique of Hirsch's Citation Index: A Combinatorial Fermi Problem" by Alexander Yong, see here http://www.ams.org/notices/201409/rnoti-p1040.pdf

Roughly: The h index is about half the square root of the number of citations...

More precisely: For an author with N citations to his paper, the h-Index always lies between zero and the square of N. Now assume that these citation are distributed on a number of papers and all partitions occur with equal probability. Then it turns out that the estimated h-Index is then about 0.54 times the square root of N. In the article it is quoted that Hirsch himself has observed that basically all h-Indices he looked at were in the range of 0.45 and 0.58 times the square root of N. Yong also states that, especially for the highly cited people in math, physics, computer science and statistics only very few people have an h-Index above 0.54 times square root of N (and they are never more that 5% above that). The article contains much more data but only for famous mathematicians such as Field's medalists or members of the National Academy of Science and it is really worth a read.

  • "Roughly: The h index is about the square root of the number of publications..." Is this math-specific? Because for CS that seems awfully low. – xLeitix Mar 16 '15 at 21:04
  • 1
    Oops, should have been "number of citations". Corrected. But then this applies universal - check the paper, it is really worth a read! – Dirk Mar 16 '15 at 21:18
  • Shouldn't it be that the h-index is roughly half of the square root of the total citations? @xLeitix One of the early and easy-to-understand discussions is here: michaelnielsen.org/blog/why-the-h-index-is-virtually-no-use It was empirically observed in physics (not math) by Hirsch himself when he proposed the index. The AMS article gives more insight into it through combinatorics and observes that (a more elaborated version of) the rule of thumb works rather well for pure math. I think it should work more or less alright for other fields if citation patters are similar. – Yuichiro Fujiwara Mar 17 '15 at 4:01
  • 1
    [ (citations)^(1/2) ] / 2 indeed seems a reasonable approximation for the h-index for most researchers I checked. For really large h-indices (> 40 or 50) the rule seems to get iffy, though. – xLeitix Mar 17 '15 at 7:03
  • 1
    The AMS article used a simple random model for the citation distribution, and gave a simple one-variable estimate (i.e., about 0.54 of the square root of the total citation count). And it turns out It doesn't seem to be far off in pure math for the cases they checked. This estimate seems to work ok for physics as well, which might suggest that how physicists get cited isn't that different than in math. In any case, if there is a typical citation pattern in a field and if you model it well, it's no wonder someone's h-index in your field typically follows this kind of one-variable estimate. – Yuichiro Fujiwara Mar 17 '15 at 8:15
1

I'm no expert on this, and this is just a comment that is a little too long to cram in the comment section.

Before addressing your question, h-index is not supposed to measure productivity alone. It attempts to measure both productivity and impact in terms of citation count. The simplest and crude measure of the former would be the number of papers a scientist has published, and one simple measure of the latter would be the total number of citations. You could say these are two crude measures of quality and quantity, although their meanings are vague. In a sense, h-index is intended to measure the quality and quantity of scientific output by a single number.

Anyway, to answer your question, assuming that you are asking if there's any serious research on how well h-index works as intended, if you just google with some obvious keywords, you should be able to find many serious academic papers that study h-index, e.g.,

A. F. J. van Raan, Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups, Scientometrics 67 (2006) 491-502.

or several papers cited in this article (pages 5-6)

or many other papers you should be able to find through usual means like checking the references at the end of a paper on the topic, looking up papers that cite them, putting more effort in googling, trying different keywords you couldn't come up with at first but you can after reading some papers on the topic, using different search engines, etc.

Note that the above parers are by no means meant to be representative examples. I just googled a bit, skimmed bits and pieces, and copy-pasted the links. As I said at the beginning of this post, I'm no expert and as clueless as you. For what it is worth, the journal Scientometrics seems to have a decent impact factor, since you seem to care about that sort of thing.

I hope real academics who work on bibliometrics chime in and give expert answers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.