Why is it bad to judge a paper by citation count?

Question

I hear a lot from the experts that citation count is a bad idea as a measure of judging a paper. This seems simply counter-intuitive to me. I would like to know if any study has been done in this direction?

Answer 1

I do not know of any study in this field, but I have a hunch that the following points apply (without any particular order, just numbered for convenience in the case of comments/replies):

1. Opportunity for citation: A paper that presents the ultimate answer to a problem is probably more valuable than a paper that presents just an unfinished attempt at solving the same problem. Yet, the latter is much easier to build upon (because it still leaves obvious extension points) and thus might be cited much more frequently.
2. Meaning of citations: There is no rule that a paper needs to be cited positively. If a paper has an obvious flaw, this may give rise to a number of other papers that just cite the flawed paper to point out what they are about to prove wrong or do better.
3. Citation scope: Referencing a paper does not mean referencing the whole paper, or its core finding. A reference to a paper might mean that the whole paper is pointed out as related work, or it could just mean that the second half of a particular statement in the introduction section is pointed out as related. A citation can well point out a remark that is completely marginal to the referenced paper, but that the citing author deemed to explain the fact in question particularly well, or that made a particularly recent mention of an old topic (to show the topic is still of interest).
4. Not all citations are equal: The question whether something is or is not cited hinges on its relative relevance compared to other citations, based on the available space left in the current paper format, not on the global importance of the work.
5. Citation habits differ: Even within one (sub-)field, citation habits differ wildly between conferences and authors. This is in part due to different paper formats and length restrictions (see previous point), but also simply because some authors are much more willing to throw in citations every now and then than others. Such habits can even vary for a single author, based on "strategic" considerations for the intended final target audience of the paper.
6. Citations can be transitive: Authors occasionally face situations where they can either list several examples of related work, or cite a single overview paper (e.g. a state of the art report), to give an overview over some basics or otherwise related topic. The latter case is clearly more space-efficient and possibly more helpful (as the overview paper is designed to give an overview while at the same time conveying the information somewhat in-depth, which often cannot be done in a single sentence of a related work section), but does not increase the citation count of the works referenced in the overview paper.
7. Papers are not atomic: There is no "natural" clustering of research questions, solutions, works and experiments. How many or how few findings go into/are allowed in a single published work is mostly a matter of style and personal preference of authors and program chairs. With that in mind, any particular finding may be presented in a single paper (which gets N citations), or split up into N papers (each of which may get only one citation). Thus, while the number of citations in this example may give a hint about the importance of each of the single written documents, it does not say much about the importance of the actual set of research questions.
8. Citation count is not normalized by field/community: As correctly pointed out by Cephalopod, '"What constitutes many depends on the field". Groundbreaking work in a narrow field might never get many citations.' Likewise, fields where publishing small bite-sized papers at a high frequency is common naturally amass a much larger total amount of citations than fields where publications are very seldom and, when they occur, cover a large amount of progress, discussion and findings.
9. Citation tweaking: As any metric, defining such a single factor for measuring performance, citation count is prone to abuse. If an author suspects citation count to be beneficial, there is an incentive to self-cite (or cyclically cite each other within an institute), and possibly even break up publications into smaller parts (related to the practice of salami slicing) to have more opportunities for those extra citations. As it can be assumed that some authors are already doing that, this renders citation count even less reliable as a quality indicator, as the "training data set" that would be used for comparison is already skewed.
10. Citation count ≠ citation count: The citation count itself is not a reliable measure to start with, as different sources will provide different citation counts, depending on what citations (and what kinds of documents that contain citations) are counted:
    • Citations in a book can be counted by chapter (point in favour of this: If each chapter is written by a different (group of) author(s), every chapter has its individual list of citations), or for the whole book (point in favour of this: A book tends to have a certain focused topic, and - more so than e.g. conference proceedings - all chapters tend to follow a common "narrative", so some central works are likely to be referenced in various chapters).
    • If possibly predatory journals are included, a variant of item 9 can ensue in which citations in less reputable sources (that are counted nonetheless) are bought, as pointed out by ff524.
    • Citation counts can include citations from totally legitimate works, which are nonetheless of a vastly different status compared to peer-reviewed publications. For example, Google Scholar sometimes seems to count student Bachelor and Master theses and other documents created while studying if the university's digital library that provides a copy of every such submitted document happens to be crawled by Google.
11. External factors influence selection of cited works: If there is one particular finding in one particular work that is appropriate to cite, there is no question that that particular paper will be cited. However, often enough, things are much more vague - there are various eligible papers that allude to the same statement in different ways, and it is not at all clear which one is preferrable to cite. At this point, external factors that are rather arbitrary can directly influence whether or not a work is cited. For instance, the older paper might be cited to show the original finding ... but the more recent paper might also be cited instead to show that the topic in question is still of interest in recent works. The older paper might be chosen because it is the original reference, or the more recent paper at the more highly-valued venue might be chosen to make the reference stronger, or yet another paper might be chosen because it contains the more comprehensible explanation. Lastly, depending on the citation style, even space may be an issue - if an author is fighting for the last few lines to stay within the allowable page limit, at least in APA style, a paper by Li et al. has an inherent advantage over one by Miller and Bowman, one by Miller and Bowman has an advantage over one by Erdenebileg and Papadopoulos, and one by Erdenebileg and Papadopoulos has an inherent advantage over a paper by Russell-Rodriguez and Pennington-Kiesewetter.

Answer 2

Citation count is a good example of a phenomenon subject to the Matthew Effect: a feedback process in which privileged individuals become more privileged as a result of their privilege. Quality, of course, does have a significant correlation with a paper drawing citations. Its citations are also strongly affected, however, by the fame of its authors, the fame of its venue, and also simply by the fact that other people cite the paper (thus making people like yourself judge it as "higher quality" and therefore more likely to cite it). Likewise, the mere fact that a paper is obscure does not mean that it is bad. Mostly, it simply means that nobody is paying attention to it. Finally, the fact that people are happy with a result doesn't necessarily make it true, as any number of scientific shifts in thinking (not to mention scandals) can attest; a paper may even draw citations precisely because other people are criticizing it.

Citation count is still a good way of getting a good first impression of which papers other people consider important, but you really need to judge the quality yourself, as a scientist.

Answer 3

tl;dr.

It only works under certain conditions.

---

There is actually a whole field of study that explores the scientific process via publication metrics, it's called bibliometrics. It is true that citation count within a field is sometimes used as a proxy to estimate an article's quality. See for example this recently published study about peer review:

Siler K, Lee K, Bero L (2015) Measuring the effectiveness of scientific gatekeeping PNAS Jan 13;112(2):360-5. doi: 10.1073/pnas.1418218112

From which the highlighted paragraphs below are copied.

Arguments in favor

First, the reasons given to consider citation count as a measure of quality (I edited the references to match the ones given at the bottom of this post):

Scientists cite work for a myriad of reasons (1, 2). However, the vast majority of citations are either positive or neutral in nature (3). We worked with the assumption that scientists prefer to build upon other quality research with their own work. As Latour and Woolgar (4) suggested, citation is an act of deference, as well as the means by which intellectual credit and content flows in science. Relatedly, we also assumed that most scientists want to produce quality work and will seldom attempt to garner credit and attention by blatantly doing bad work. Thus, on the whole, the attention and impact associated with citations provides a reasonable measure of quality. Citations provide an objective and quantitative measure of credit and attention flows in science.

Arguments against

They also discuss the limitations of using citations as a measure of quality, the logarithmic distribution of citations:

Because citations are often distributed exponentially, with a few articles garnering disproportionate attention (5), we also used the logarithm of citation counts as a dependent variable to diminish the potential influence of a few highly cited outlier articles

And the 'social status' effect:

scientists often rely on heuristics to judge quality; status of scholars, institutions, and journals are common means of doing so (6, 7). Unsurprisingly, citations received by manuscripts were positively correlated with the impact factor of the journal in which it was eventually published.

In short, comparing citation count of articles published in journals with a very different impact factor is a bad idea.

Pages 46 and 47 of reference number 3 give an exhaustive list of limitations of citation count as a proxy for quality (I edited the list for the sake of brevity):

• Time-dependent factors Due to the exponential increase in scientific output, citations become more probable from year to year.
• Field-dependent factors Citation practices vary between science and social sciences fields [...] and even within different areas (or clusters) within a single subfield [...], small fields attract far fewer citations than more general fields [..].
• Journal-dependent factors [...] journal accessibility, visibility, and internationality [...] as well as the impact, quality, or prestige of the journal may influence the probability of citations.

• Article-dependent factors [...] There is also a positive correlation between the citation frequency of publications and the number of co-authors of the work [...]. And, as longer articles have more content that can be cited than shorter articles do, the sheer size of an article influences whether it is cited [...].

• Author/reader-dependent factors The language a paper is written in [...] influence the probability of citations. [...] citations are affected by social networks: authors cite primarily works by authors with whom they are personally acquainted. [...] men receive substantially more citations to their work than women do.

Many regular users of this site will love that one:

• Availability of publications Physical accessibility [...], free online availability of publications [...], and the publishing media [...] influence the probability of citations.

And lastly:

• Technical problems [...] The incorrect citing of sources is unfortunately far from uncommon: Evans et al. (1990) checked the references in papers in three medical journals and determined that 48 percent were incorrect: “The data support the hypothesis that authors do not check their references or may not even read them” [...] In a similar investigation, Eichorn and Yankauer (1987) found that “thirty-one percent of the 150 references had citation errors, one out of 10 being a major error (reference not locatable)” [...] Additionally, problems stemming from homographs and synonyms can arise when researching publications and deriving citations from citation databases using authors’ names.

All these are linked to plentiful citations that you can find in the original document available for download here.

References

1. Lynn FB (2014) Diffusing through disciplines: Insiders, outsiders and socially influenced citation behavior. Soc Forces 93(1):355–382.

2. Hargens LL (2000) Using the literature: Reference networks, reference contexts, and the social structure of scholarship. Am Sociol Rev 65(6):846–865.

3. Bornmann L, Daniel H-D (2008) What do citation counts measure: A review of studies on citing behavior. J Doc 64(1):45–80.

4. Latour B, Woolgar S (1979) Laboratory Life: The Construction of Scientific Facts (Sage, Los Angeles).

5. Lotka AJ (1926) The frequency distribution of scientific productivity. J Wash Acad Sci 16:317–323.

6. Long JS, Fox MF (1995) Scientific careers: Universalism and particularism. Annu Rev Sociol 24:45–71.

7. Lee CJ, Sugimoto CR, Zhang G, Cronin B (2013) Bias in peer review. JASIST 64(1):2–17.

Answer 4

First off, don't trust generalizations too much. The number of citations is not a perfect measure by far. First, more citations take time to accumulate, once could possibly look at citations per year or something similar. Second, it is possible to cite your own work so without filtering out so-called self-citations you may see inflated values. Self-citation, in itself is not necessarily an evil either, there are many reasons why one must reference ones own earlier work. One obvious reason is that later work often builds on earlier work and part of that is usually earlier work by the same person. Third, the number of citations are field dependent and in bibliometrics methods exist to remove such bias. Hence an article in a hot topic with much research will receive more hits than an article, no matter how excellent, in a small field.

So, when judging an article from its citations, it is useful to keep the problems in mind and not over-interpret. This is essentially not different from any normal approach to any data set.

Answer 5

The reason citation counts are not good for judging paper quality is that there are a great many reaons (15, 28, 26 - depending on the study) for citing, and citation practice vary between disciplines. See:

Case, D O & Higgins, G M 2000 How Can We Investigate Citation Behavior? A Study of Reasons for Citing Literature in Communication. JASIS 51(7) 635-645

for references to earlier studies, and report of a study in the discipline of communication.

Answer 6

Others have mentioned, but it bears repeating - citations are not necessarily positive. Plenty of papers use faulty methodologies and are cited as examples of what not to do.

Also, at least within the social sciences, citation is correlated with age and 'first mover advantage'. Because any academic paper worth its salt is going to cite previous work on the topic in the literature review, older papers will naturally garner higher citation counts, all else held equal. This has very little to do with the quality of those papers.

Answer 7

There have been many good answers to this question, but I wanted to frame what I see as the most important points.

Citation counts measure impact not quality: The citation count of an article is a measure of research impact, and not quality. Even as a measure of research impact, it has issues, but in general, "research impact" is closer to what it represents. A high quality new paper will have minimal citations, because it has had minimal time to have an impact. Similarly, the quality of a paper is only loosely related to research impact. Some high quality papers are difficult for the literature to digest. Some low quality papers evolve into standard publications that are commonly cited in a given context. Entire literatures may use questionable methods but build a whole ecosystem of mutually reinforcing citations.

Citation count is confounded by time since publication: One of the biggest points is that: citation_count = average_citations_per_year x years_since_publication

Time since publication is unrelated to quality, yet it is one of the main drivers of article citation count. Thus, even if you did see citations as an index of quality, average citations per year would be a better index than total citations. Even there, it gets complex because articles in a given field tend to have a general distribution of citations over time where for example, citations per year typically tends to peak early and then gradually declines over time. So if you were trying to get a pure measure of quality based on citations, you would probably try to estimate the expected citation count after 30 or more years based on how many years has passed and known citation distribution characteristics.

Assorted other points:

• Fields differ in citation practices: Some fields have short citation half-lives so that more of the total citations an article will receive typically accrue in a shorter period of time (e.g., 5 years versus 15 years). Some fields include more references and therefore there is a greater sum of citations in the system. In general, this is more of an issue if citations are being used to do comparisons across disciplines. If you operate in a particular field, then this is less of an issue. And if you are aware that STEM fields tend to have shorter citation half lives and more references relative to social sciences, then you can incorporate this into your perspectives.
• Self-citations can be removed: Many systems like Scopus allow for the removal of self-citations. This can be useful where you suspect such gaming is occurring (or generally where there is an academic that publishes a lot and self-cites a lot without getting citation traction with other academics).

Using article citation count as an index of quality

While most answers here point out the problems with citation counts, it is worth taking a balanced perspective. After five or ten years after publication, average citations per year provides some information regarding the quality of a publication. Actually, it is a measure of impact potential as indexed by achieved impact over a finite period of time. It's not definitive, but it is a useful bit of information. If you want to assess quality, either you or someone suitably skilled needs to read the article and appraise it.

In particular, if after five or ten years, an article has almost no citations, then this is a bit of a red flag for the quality of the paper. Likewise, if an article has hundred of citations per year, this suggests that it is likely to be an important piece of work. None of this is definitive, but it is suggestive and useful information.

General points on how to use citation counts

• Citation counts are a measure of impact, not quality
• Average citations per year is a better measure of the "impact potential" of a paper than total citations
• Citation counts become more informative as more time since publication accrues. As a rough rule of thumb, five years post publication is probably a rough minimum to get a somewhat accurate sense of citation potential.

Answer 8

First of all, what does it mean to better?

I know this may sound like nitpicking, but I can provide an example that gives a strong negative correlation between citations and a certain measure of better. I typically write algorithms for my research. Generally speaking, you don't get to publish unless your algorithm is an improvement in some meaningful way over previous algorithms. But you certainly need to cite previously published algorithms! Thus, for a given problem, the early algorithms will have a very large number of citations, while the later algorithms will have a much smaller number of citations.

So if your definition of "better" is that the algorithm is faster (a very reasonable definition), then the papers with large numbers of citations are likely to be the worst in the literature (since they are the oldest).

As an anecdote, I just published a paper on an algorithm that I have reason to believe will close the book on optimization for that particular problem (partly because it's really fast, and partly because it's not a very hot topic). As such, I don't expect that paper to get many citations, even when compared with previous algorithms on that same problem.

Answer 9

There was a paper published in 1994 in a medicine journal entitled A mathematical model for the determination of total area under glucose tolerance and other metabolic curves which describes a method for computing the area under a curve, dubbed "Tai's method". This paper got 311 citations according to Google Scholar, which isn't too bad for medicine papers apparently (although some can rake in literally 1,000 citations...).

This method is the trapezoidal rule and according to Wikipedia, it was known to ancient Babylonians in 50 BC. Every undergrad with a vague math education, perhaps advanced high school students, or someone who reads a pop math blogs, knows about it.

You wouldn't know this insanity from just looking at the citations of the paper. There are many good arguments in the other answers, but if this isn't damning enough to make you look at citation counts with some circumspection, I don't know what will.

Answer 10

On top of all the other good points made already, there is also Goodhart's law to consider:

A measure ceases to be a good measure once it becomes a target.

Answer 11

I think it boils down to the question:

Do you use citations as a measure or as a metric.

A measure is a value you get by measuring a quantity (e.g. miles, seconds, citations, words). A measure is a mostly objective attribute. A measure does not have any meaning, it is only a quantisation.

A metric uses usually multiple measurements and combines them. After the combination the metric can be used and compared to a set of rules or other metered objects in order to sort or value them. The combination of measurements is the step were a) meaning and b) subjectivity enters the whole process.

It is therefore absolutely understandable to use citations as a measure but you will most likely never use it as a metric itself. When the number of citations is used as a metric for paper-quality, assumptions need to be made which cannot be proven (except by measuring something else as well).

Only because one can measure something does not mean that the measurement contains any information.

Answer 12

Doe, Umanskaya, Milton and Bennt suggest in [DUMB 1999] that it is indeed acceptable to evaluate papers based on their citation count, and that paper of theirs has over 5,000 citations, so we can probably take their word for it.

Answer 13

Human beings, regardless of intelligence level, generally tend to subscribe to the "herd mentality". I guess it's because we, as animals like any other animal, want to conserve energy. For example, if everyone believes Stephen Hawking or Lisa Randall is rarely wrong, then it's in my best interest to focus on what he/she has to say. It will save me the energy of exploring other sources and finding answers for myself.

At someone's request I will try to clarify my answer.

Let's say I'm writing a paper about black holes. Obviously, I want to write the best possible paper but I'm not the best physicist on planet Earth. The best physicists on planet Earth will probably have the most citations on the subject matter. Does having the most citations about a subject mean that the highly cited individual is always correct concerning the subject? No. I have to realize that if I'm referencing that person's work. Assuming that Stephen Hawking is right all the time about black holes or even assuming that he is right most of the time is a huge and dangerous assumption.