15

Impact factors and citation counts are widely used metrics to judge the reputation of researchers. However, these metrics can be misleading, and in some cases can be manipulated.

Why do universities place so much importance on citation counts and accumulative impact factors?

15
  • 18
    Why do you think a "majority of universities" really care much about this? Citation needed.
    – Buffy
    Nov 25 at 16:04
  • 4
    You mention exceptional cases. I doubt that promotion was due only to "bean counting", though it may have alerted people to take a look. And "majority" seems far too strong.
    – Buffy
    Nov 25 at 16:32
  • 4
    In A number of European countries, you can't get a PhD unless you've published x so many papers with impact factor higher than y. I know that in Spain, application forms for postdocs and faculty jobs specifically ask for impact factors. In the UK the research excellence framework ranks the quality if papers and decides university funding. Its claims not use IF (but will use citation counts), however university planning for REF generally does use IF. My academic probation criteria said i needed 1 paper IF>10 or 2 papers > 5. Nov 26 at 7:32
  • 5
    I hate to say it... but it's because the people who set these KPIs tend to be dumb business types rather than academics who have no idea how to effectively run a research institution Nov 26 at 13:53
  • 4
    @IanSudbery They must have different standards for different departments, because I don't think my field has any journal with IF > 5.
    – Kimball
    Nov 26 at 14:18
33

Because it's easy

Let's say you got 100 researchers to evaluate. How would you go about doing it? You could read their papers in sufficient detail to understand them (would take months or years, especially if they're not in your field). You could rely on peer evaluations and recommendation letters (already done extensively). You could look at prizes and awards won (also already done extensively). Or you could look at citation counts and impact factors. The last is easy to do, and while it's not perfect, it is objective and there is a correlation with research quality (citation counts is the standard method to measure excellence in bibliometrics).

Besides, many non-specialists care about citation counts and impact factors, e.g. all the major university rankings use it, and students look at rankings. Doing badly on those rankings can make one less attractive to students, with all its negative consequences.

See also this:

But. Faculty candidates are necessarily judged by people who are not experts in their field. Without the expertise to judge whether your work is really good, those people must look at secondary data that correlate strongly with successful researchers. One of those secondary characteristics is "pedigree". Did you get your degree at MIT, Berkeley, Stanford, CMU, another top-10 department, or somewhere else? (What's an "Ivy League"?) How good/famous is your advisor? If they're really paying attention: Where did your advisor's other PhD students get jobs, and how well are they doing now?

Fortunately, most good departments do make a serious effort to understand the quality and impact of applicants' results, instead of relying only on secondary data. Also, secondary data matters considerably less once you actually have an interview.

1
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – cag51
    Nov 28 at 22:50
11
  1. Because many funding agencies give a lot of weight to citations and impact factor,
  2. Because many ranking agencies (of all sorts) will give higher rankings to institutions with research-active and well-funded faculty, as measured by such metrics,
  3. Because stating in a recruitment brochure that your faculty publish on average so-many papers a year will make you look better than so-few papers a year.

Yeah it’s all marketing, but in these days where governments are funding a decreasing proportion of the cost of education, where universities compete for students, and where the ratio of administration to faculty keeps increasing, the administration needs new revenues. See this article in Forbes magazine for additional context and financial pressures on institutions.

4
  • 1
    I agree with you its all because of faulty methodologies adopted by funding agencies, but academia didn't resist rather becoming part of it.
    – Mohaqiq
    Nov 25 at 18:32
  • 2
    @MBK it is a self-evident truth that most universities are perfectly happy to just “follow the money”… principle of least resistance. Nov 25 at 18:46
  • Question: "Why do we give so much importance to citations and impact factor?"; Answer: "Because some people give so much importance to citations and impact factor". OK, but this just displaces the question a bit, it doesn't really answer it.
    – Stef
    Nov 26 at 12:23
  • @Stef no and yes. If the people who give such importance hold the $$ then yes it does place the ball in their court, but they would argue it’s their preferred way of allocating resources. Would you rather send your kids to a top-ranked school or a lower-ranked one (cost notwithstanding )? Nov 26 at 13:12
3

Question is impossible to answer since the premise is false, or at least has no proof. Specifically,

why majority of universities give too much importance to citation count and accumulative impact factor

No university I am aware of cares at all about impact factors, which are in most cases fake (the ones reported by journals).

Citation count is slightly more important, and is used at times as a proxy of "impact" of a scientist, but even this is not universally recognized, and most universities do not explicitly check this measure.

In fact, promotion and hiring is based more on:

  • reference letters and peer evaluation
  • places of publications: prestigious or not.
  • acquaintance of existing faculty members with the candidate
  • number of publications in selective places (not number of citations)
11
  • 2
    In the US, at least, promotion is largely a result of peer evaluation and peers have a more sophisticated view of the accomplishments of a candidate. True, the advice of a promotions committee isn't the final word, but is nearly always respected - or chaos results.
    – Buffy
    Nov 25 at 18:16
  • 3
    "impact factors, which are in most cases fake (the ones reported by journals). Citation count is slightly more important, and is a proxy of "impact" of a scientis" Your these sentences are contradictory. Citations determines the impact factor. So, if impact factor is unreliable, then how citations can be a reliable metric? As I mentioned in my previous comment publishing 100s of articles in a year cant be justified so as the citations of such researchers.
    – Mohaqiq
    Nov 25 at 18:30
  • 1
    @MBK, I didn't say citation counting is "reliable", I said it's more important in the process of being promoted, because it's something that people and committees and peers look at. No one I know of looks at "Impact factors", and moreover IFs stated by publishers are unreliable (or "fake").
    – Dilworth
    Nov 26 at 1:53
  • 1
    @ZeroTheHero Predatory journals are irrelevant. All citation counts, impact factors and h-indexes are taken directly from WOS and most predatory journals never make it there. If they do, they remain somewhere at the bottom of the ranking with low IF. They generate few citation from other WOS journals. Only WOS to WOS citations count.
    – Vladimir F
    Nov 26 at 7:57
  • 1
    @VladimirF WOS is not the sole supplier of metrics (there are enough questions here in GoogleScholar for you to know that). My point is that the total number of papers is now no longer accepted as a metric. Nov 26 at 13:07
2

I have no idea about "majority" either, but we are occasionally discussing what a "fair metric" would be for counting publications. Everyone agrees that the evaluation of the true scientific merit is impractical (or even impossible) for a variety of reasons from impossibility to carefully read and evaluate all papers in a variety of completely different areas to the mere fact that to estimate a true impact of something, one has to wait for 10+ years at the very least.

Our current system (as far as I remember it) is to have tier 1 and tier 2 journals with 2 points for a publication in a former and 1 in the latter (there are also some fancy rules about sole publications vs. co-authored ones I was too lazy to read properly, so I'd rather skip them here). It is far from ideal and is constantly critiqued from all sides, but it is a not too bad compromise after all. I've heard of much worse arrangements where the formalization was brought to something completely absurd (from my perspective, at least). The tiers are determined by "general agreement" (and there is some fight here) with the impact factor playing some role, but not the decisive one. In general, it is believed that the individual citation metrics (like h-index) reflect the true impact of a researcher much better than the sum of journal impact factors times the number of papers in that journal.

The main reason for the very existence of this point system seems to be that neither we would like to explain to the administration what is worth what on the routine basis, not the administration would like to hear such explanations more often than absolutely necessary, so everything should be reduced to a single number. When really important cases arise (like promotion to tenure, etc.) the number merely needs to be above a certain (fairly low) threshold and the decision is made based mostly on the recommendation letters and such, but for trivial issues like annual merit salary raise that number suffices and everybody can see how the awards are made. Transparency here is at least as important as fairness and since the latter can be hardly achieved even in theory, resorting to the famous KISS principle is a reasonable idea. As long as the point system (whatever it is) doesn't acquire life and power of its own that overrides common sense, it is harmless and even useful.

Of course, the situation varies from place to place, but what should be remembered is that many rules like that are not creations of the universities, but rather of the faculty in the particular departments, and the ugliest monsters are born when people do not want to agree on anything or even to listen to each other. Various "rankings" by independent agencies also play some role, of course, and the citation factors play noticeable role there, sometimes in a rather weird (again, IMHO) way.

As I said, the current system is far from ideal, but if you ever come up with a bright idea of a really good way (fair, transparent, and practical), let me know and I'll be happy to advocate for it locally :-)

2
  • Nice points raised, but this is not an answer to the question.
    – Dilworth
    Nov 26 at 1:56
  • @Dilworth Agreed, but, as it has been already said in the comments, the very premise of the question is "questionable", so I just described what is going on at one particular place I can speak about with some certainty. The point is that we do put some importance on those metrics (and I tried to explain why) though not so much. If somebody works at the place where "so much" is a rule, they might be able to provide more relevant information, of course.
    – fedja
    Nov 26 at 4:11
1

A major issue in my view is the concept of "objectivity" and the issues with it. Objectivity can mean many things, but often it is taken to be the result of a transparent and reproducible measurement procedure, and associated with the removal of the influence of personal interests and opinions.

The essential dilemma with this is that the ability and potential of a researcher and the value of their work are so complex (and surely not one-dimensional, as any ranking needs to be) that chances are that no "objective" measurement procedure can do them justice. In some places there is a tendency to prioritise straight numerical measurements despite being obviously deficient (I'm not necessarily saying the citation counts and impact factors are the best of those). In some other places decisions are made in intransparent manners by commission members, taking into account reference letters and other information from peers. This can work well, but also is connected to favouritism and old networks protecting their field.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.