23

Academic indicators (h-index, impact factor, modified h-indexes, etc) have a long string of criticism by academics and not (see here, here or here).

Instead of debating (endlessly) on how "not representative" and flawed these numbers are (or howling generic rants...), I would be interested to know if Academia.SE community members have objective facts and reports on how academic workplaces are currently using these indicators. For instance in the upcoming REF2014 (UK), academics in my institution are urged to use, as their contributions, the papers accepted in highest impact factors journals.

In summary, can you trade your h-index for a better paying job?

Improve this question
6
  • 7
    For a better paying job, one might rather leave academia altogether, I don't think the oil industry or the arms industry or other evil, rich industries care about the h-index ;)
    – gerrit
    Oct 25, 2012 at 16:01
  • i'm convinced that academic jobs are moving towards bonus, company-style approaches, as @walkmanyi says below. So then there'll be no need to switch from academia to oil companies :)
    – ElCid
    Oct 25, 2012 at 19:07
  • 1
    Do you have a reference for REF? We are explicitly being told quality is not IF.
    – StrongBad
    Oct 25, 2012 at 19:50
  • @ElCid If people want to earn a lot of money, I'm sure the oil industry and the arms industry will offer a lot more than academia regardless of the system...
    – gerrit
    Oct 25, 2012 at 20:17
  • @DanielE.Shub we had an internal discussion on the fact that REF2014 will not be based on metrics. But then a good point emerged saying that it will expose the exercise to subjectivity, which is clearly worse. So an internal communication was sent to urge academics selected for REF to pick their choices among the ones with higher IF
    – ElCid
    Oct 26, 2012 at 13:43

5 Answers 5

Reset to default
10

On the European research market, various opinions have been offered on the use of h-index by ERC grant evaluation committees, and the broader relationship between h-index and ERC funding success. People who provide such opinion can be classified in three categories:

  • affiliated with the ERC
  • consultants whose business is to offer advice to (potential) candidates
  • individuals involved in another way, whose advice is most probably only anecdotal

So, what do the first two categories have to say? The official word from the guidelines is, well… absent. But it is politically correct to assert that h-index is not a good indicator of scientific quality and, as such, not used. This gives quotes like:

Quality in science is not proved by accumulating quantitative points. The role of commercial impact factor and h-index is limited. Overemphasizing of publish or perish policy leads to a gradual perishing of all.

On the other hand, practical advice might be a bit more nuanced:

There will also be a new h-index study the successful 2011 awardees in the PE and LS domains. The h-index is regarded as a background indicator rather than a determining factor and the study on the 2010 Advanced Grant awardees showed that each panel made awards across a considerable range and that there were significant differences across the different ERC panels. There was a big variation between different disciplines within the main domains.

and this:

Every applicant had to choose his 10 best publications published in the recent decade and add how many times each of these papers were cited in the literature. The total number L of these citations describes well how the scientific community perceives their recent achievements. These numbers were provided by the ERC in the dossier of every applicant. However, during the evaluation process the panel did not put much emphasis on any bibliometric data. It was the opinions of the experts which did matter, not the bare numbers. Only after completing the evaluation process, I realised a correlation between these data and the final outcome.

To give another perspective: in France, a new evaluation system for higher education and research was put in place 5 years ago (the newly created agency performing the evaluation is called AERES). AERES evaluates each research group every 4 to 5 years, in order to give it an overall rating, which could be A+, A, B or C. This had at least two very practical consequences that I know of:

  • For yearly financial negotiations between each university and the Ministry for research, the ministry started to require a spreadsheet with the number of university teams rated A+, and the number of A team (B and C didn't seem to count). Financial support was then dependent on that number, at least as a starting point for the negotiations.
  • It became customary to include this grade in your French grant applications, because a A+ rating was considered a serious advantage. This was written in the “rules”, however…

So, all in all, are bibliometric and, speaking more broadly, academic indicators really adopted by institutions? Hell yeah! They make deciders’ job easier: quantification of research quality makes it easier to make decisions. Smart decision makers do realize, however, that a single indicator does not make for good decisions.

Improve this answer
3
  • Interesting. Could you be more specific on which basis h-index should be officially calculated in those contexts you mention? I mean, e.g., for a computer scientist WoS is not really a good (favourable?) source of bibliometric data as conferences tend to be preferred in many sub-communities. On the other hand, e.g., Scopus is of a lower-profile and Google Scholar and in turn the Harzing's PoP based calculations are more "anecdotal", rather than "officially useful" as they include also self-citations and less-prominent publications (e.g., TRs).
    – walkmanyi
    Oct 25, 2012 at 15:09
  • For computer scientists, all methods of computing h-indices are unreliable. But they're presumably unreliable in the same way for everyone in the same subfield, so they can still be used for comparative purposes. As with any unreliable statistic, it's useful to take multiple measurements, so using the h-indices from WoS and Scopus and Google Scholar and Microsoft Academic Search is probably the least worst option.
    – JeffE
    Oct 26, 2012 at 14:51
  • 1
    (Arguably, there's no reliable, objective way to compute h-indices in any field. Who decides which citations from which journals count, and how? Fortunately (sic), the reliability of citation statistics is demonstrably lower in CS than in other fields.)
    – JeffE
    Oct 26, 2012 at 14:55
6

In Czech Republic, researchers are obliged to indicate their most important publications, together with the number of citations, the corresponding impact factors of the journals, as well as their own WoS-based h-index on major grant proposals to national grant agencies (GACR, TACR, etc.). In a consequence, universities care for h-index and other citation metrics internally as well and in result citations/impact factors are tracked and form a basis for annual evaluation, possibly even leading to end-year bonus calculation. So yes, at least in Czech research space citation metrics, such as h-index and impact factors are a big deal.

Later edit: Since you ask for objective evidence, I refer to the recent manual for GACR standard projects starting in 2012, paragraph 4.2.10, b-d (all in Czech).

Yet another edit in response to ElCid's comment: There was an equation for calculating an extra-ordinary bonus taking into account the impact factor of the journal authors get their paper accepted in and then taking into account the stated (percentual) contributions of each individual co-author. In result, the formula spit out the amount of money each department-resident co-author would get for the paper. The bonus was a department-specific policy.

Secondly, there was an annual evaluation taking into account the number of papers produced by the researcher, the number of citations received in that particular year (sometimes extremely hard to track), impact factors for journals of the papers concerned (both submitted and those receiving citations) grants received, and other minor factors as well. The evaluation was a faculty wide policy, I am not sure whether it led to direct financial benefit to the researchers, but certainly these metrics were important in the internal university-wide division of funds which also partly hinged on aggregates of the above described metrics. To my understanding, these policies formed an incentive for the faculty to target high-impact journals in their respective fields.

Improve this answer
2
  • Could you describe the bonus? Are research-active staff expected to get it anyway? What are the effects on other staff?
    – ElCid
    Oct 25, 2012 at 14:21
  • I'm really enthused by this answer... I am looking forward to other countries sharing and adopting this model, which I believe is ready to deployment, given the pressure and the tight deadlines dropped on academic staff
    – ElCid
    Oct 26, 2012 at 13:13
5

The main reason to look at impact factor (at the schools where I've worked*) is so that a future hire can most probably obtain funding to continue research (support grad students, purchase equipment, etc.).

Quality publications become an important aspect of any research proposals a future hire would submit. Research proposals are evaluated by other researchers within the network of the research program. Those researchers can freely use impact factor of a proposal's publications as a criterion for evaluation. I don't know of any programs that use it as a mandatory criterion, probably because it's still controversial.

Some professors might get hired because they already have good funding, sometimes because of R&D contacts with industry, a good IP track record (patents), and a low impact factor may be a terrible reason to reject a candidate like that.

As for converting h-index to better pay, I'd say if you're at the right institution, you can convert it to a better package (salary is only one aspect). Impact factor is not the only indicator that you will be successful, but might be an important one for tenure-track (assistant) professor positions. It may not get you a higher salary, but it probably will get your CV higher on the list.

*Hiring committees are formed and they are free to set the criteria for selecting candidates. Impact factor has been used on several committees at my institution, but it's not an established policy. I believe this is an issue of academic freedom.

Improve this answer
5
  • 2
    Research proposals are evaluated by other researchers within the network of the research program. Those researchers can freely use impact factor of a proposal's publications as a criterion for evaluation — Presumably researchers within a grant applicant's own research field can judge the quality of that applicant's research directly, no?
    – JeffE
    Oct 25, 2012 at 22:55
  • @JeffE If you're fortunate enough to have those experts evaluate your proposal, then I agree. In some emerging fields of research, I'm not sure proposals always wind up in the hands of committees of experts. In such cases, a high impact factor is probably important.
    – Fuhrmanator
    Oct 26, 2012 at 0:11
  • 2
    @Fuhrminator: Wait, I'm confused. For papers in an emerging area with few people qualified to judge the quality of the work, most citations necessarily come from authors who can't judge the quality of the work. Why would anyone trust such citations as an indicator of quality?
    – JeffE
    Oct 26, 2012 at 15:04
  • @JeffE I didn't catch your last comment since my user id is different. I was thinking of young profs in an emerging field who submit a grant proposal to a govt funding agency. The reviewers, none of whom are experts, might use impact factor for lack of anything else. They could also write off the research as bunk because it's some new fad. I'm in IT in Canada and federal grant proposals don't always wind up in the right evaluators hands. They are volunteers, and try to do the best job, using whatever tools are available to evaluate concretely.
    – Fuhrmanator
    Jul 26, 2013 at 18:15
  • The reviewers, none of whom are experts, might use impact factor for lack of anything else. — I suppose they might, but I've never seen that actually happen, either in hiring committees, tenure committees, or grant-review panels. I have, however, seen entire fields written off as pointless fads, despite high impact factors.
    – JeffE
    Jul 26, 2013 at 21:48
5

As in JeffE's comments to another answer, it is very unclear to me that anyone should be happy that "bibliometrics" assume official stature, especially in the particular situation of "emerging" fields, where one's livelihood thus becomes contingent on opinions or behavior of non-experts?

In the U.S., in mathematics, it seems that this official stature is very recent, in contrast to various places in Europe where (apparently) the bureaucracy was even less shy than here about insisting on "simple" quantification of the "performance" of academics and departments. Of course, presumably, administrations have always simplified their private appraisals of departments and individuals for purposes of "decision-making" (a.k.a., deciding who gets the money), but more recently commercial products (from our buddies the traditional publishers) have been promoted to university administrators, and have been bought and paid for, over the public objections of faculty...

One point is that traditional publishers are happy, I'm sure, to have the significance of their gatekeeper "peer-reviewed" publications more firmly ensconced by the effective endorsement of their "rating" software packages. "Conflict of interest" comes to mind, for one thing. (But I'm not eager to trade this commericalized U.S. manifestation for the systematic, nation-wide version available too often in Europe.)

The real problem is that this commodification of "research/scholarship" adds a function-less layer of misdirection and noise to an already challenging enterprise, with already-precarious economics. For those who wish it to be a "revolution" that gets us out from under some old regime, I fear that instead it is merely a different incarnation of the same thing, sometimes owned by the same mulit-national corporate entities.

Improve this answer
2
  • 1
    How does this answer the question of how broadly these indicators are actually used by institutions?
    – F'x
    Oct 27, 2012 at 19:37
  • Specifically, in the U.S. this is a relatively new thing, but is being put in place as a definitive metric in the eyes of administrators. I know people in the UK and elsewhere in Europe (in math) who are precisely judged by such metrics.
    – paul garrett
    Oct 28, 2012 at 0:15
1

In many places of South Korea, IF is definitely used for job performance evaluation. In one Research Institute I am aware of, there is yearly evaluations which are quantitative, giving each person a score out of 100. The actual impact factor of each journal publication is used in the calculation, such as the highest IF journals are given 3x more points than low or non. The final score determines job promotion or firings.

Improve this answer
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .