Using the h-index to compare researchers from different fields

Question

We are all aware that the h-index of a researcher is the metric for evaluating the cumulative impact of his/her research publications. I am asking the question regarding a doubt that arises when we compare researchers based on the h-index.

I come from a background of theoretical physics where we come across two types of researchers: (1) those who works on theory and (2) those who works on data analysis (mostly within a collaboration). For example, consider the following two researchers:

1. One is a senior professor who works on quantum field theory and string theory and has produced several outstanding papers over 25 years. His total citations = 7311, h-index = 40, i-10 index = 57.

2. The second has obtained his PhD in gravitational wave astronomy about 3 years ago and is presently a postdoc at Max Planck Institute for Gravitational Physics. Most of his papers involve doing data analysis as part of the gravitational waves collaboration. His total citations = 31121, h-index = 46, i-10 index = 75.

So, in my opinion, it wouldn't be a wise idea to compare them based on their citations and h-index since the more experienced researcher has a much lower h-index even though he has produced several outstanding papers in his area of expertise.

My Question:

In view of the above example:

• How do you compare the quality of two researchers when they are in different fields of research? Are there any other means to evaluate the quality of research?
• How do you evaluate researchers where most of their citations come from papers where they are one of a large number of co-authors and their non-collaborative works are cited much less often?

Answer 1

Reliance on any single index has many confounds. And these flaws are amplified when you attempt to compare academics across disciplines.

That said, there are more thoughtful ways to make use of metrics.

Here are some of the main things that should temper your use of h-index:

• Proportion of first-author or lead-author papers (e.g., 1st, 2nd, last author): All else being equal, the esteem attributable to an author in publishing an academic paper is increased when the author leads the work.
• Number of authors per paper: All else being equal, fewer authors per paper suggests that the author of interest is contributing more to each paper.
• Average time since publication: Papers accumulate citations over time. So, an academic with many recent publications may have many papers that will ultimately contribute to their h-index after a period of time has elapsed for those citations to accrue the requisite number of citations.
• Discipline-specific citation practices: Disciplines vary in their referencing practices. The two big differences are (1) number of references per paper, and (2) citation half-life, which is to say the time it takes an article to receive half of all the citations it will ever receive. People who are in fields with more references per paper, get more citations and as a result, a higher h-index. People in fields with shorter citation half-lives see a more rapid rise of their h-index, although eventually (10 to 15 years after publication), differences in citation half-life will generally matter much less. In addition, citation databases vary in coverage of different fields (e.g., web of science probably makes biomedical researchers look much better than computer scientist compared to Google Scholar).

What are you trying to measure? Here are some fairly orthogonal elements from which you can conceptually derive other indicators:

• Annual Output: Their contribution in partnership with others to Annual Academic Output, where output corresponds to overall value: some kind of product of both quantity and quality (or impact).
• Personal Contribution: The proportion of their contribution to that output.
• Career length: The number of years that they have been making this impact.

So, from this perspective, h-index is mostly a function of annual output, career length, and the vagaries of discipline-specific citation practices.

Personal Contribution Indicators: If you are interested more in personal contribution, you might get some of the following indicators:

• Number of first-author and lead-author papers. This could be broken down based on the quality of the output as indexed by things like discipline specific journal ranking (e.g., Q1 on Scimago) or annual or total citation counts.
• Fractionated paper or citation counts. E.g., sum of papers where the value of each paper is one over the number of authors. This would mean than one sole-author paper equals 10 10-author papers. Other weightings are possible to incorporate the value assigned to first-authorship or incorporate the assumption that actually having 10 10-author papers is more valuable than 1 sole-author paper.

Productivity indicators:** If you are comparing researchers at different career stages, it can be important to control for the fact that people with longer careers have had more time to publish and more time for those publications to get cited.

If you are more interested in research productivity, then you probably want to focus on indicators of either average annual academic output or output over some recent time period (e.g., last 3 years, last 5 years, etc.).

A few indicators include the following:

• H-index divided by years since first publication. Variants include the annual rate of increase of h-index over the last five years. Note that this does not control for variation in personal contribution or discipline citation practices.
• Number of first or lead author papers per year. This could be adapted to focus on number of such papers in journals of a certain quality.
• Average annual increase in annual citation counts over the last x years.

Summary: All citation-based metrics have issues. That said, if you are aware of the limitations of the different metrics and draw on a complementary set of indicators, you are more likely to make a considered assessment of an academic's research impact.

In general, it makes most sense to compare researcher metrics to discipline-specific norms. It's important to characterise their publication style: quality versus quantity, degree of involvement in each paper, small number of authors per paper versus many authors per paper, etc. And it's also important to be clear on whether you are interested in recent productivity versus total career output versus total career impact.

More importantly, metrics are heuristics. Where important decisions are being made, they shouldn't replace reading the person's actual work or seeking out assessments by knowledgable experts.

Answer 2

Seems like relatively easy problems to solve.

How do you compare the quality of two researchers when they are in different fields of research? Are there any other means to evaluate the quality of research?

Compare to the median in their field. For example, search up the metrics of ~10-20 researchers in the first field and calculate their median h-index. You can also calculate the standard deviation. Then you can place the researcher into a percentile, and compare that.

How do you evaluate researchers where most of their citations come from papers where they are one of a large number of co-authors and their non-collaborative works are cited much less often?

You can exclude outliers by removing them from your sample before calculating the h-index.

By the way, there's a relevant section in the Wikipedia article on h-index you might be interested in.