Does Impact Factor reflect the quality of a journal?

Question

Is the impact factor really useful for judging the quality of a journal article?

Answer 1

No.

---

To answer Paul's comment:

The title of the post asks about journals. For a well-worn list of criticisms of Impact Factor as an indicator of journal quality, see the Wikipedia article and the sources it cites. In many disciplines, important papers receive most of their citations well outside the IF's two-year window. Raw citation data can be manipulated by editorial policies, some more nefarious than others, or even by individual papers. Thomson-Reuters' calculations of Impact Factor are not reproducible, even using their own citation data. Et cetera ad nauseam. But most importantly: Having lots of citations is not the same thing as quality.

The body of the post asks about journal articles. Impact Factor is (roughly) the average number of citations to all papers published by a journal in a given time window. Even if it were a reliable measure of average quality (which it isn't), it would say nothing at all about the quality of any individual paper.

Answer 2

If the other two answers haven't made this obvious. The answer is no, but it can be taken even further than what the above answers suggests. We can study this!

The above image is the coefficient of determination between the impact factor of journals and the 2-year citation rate of their papers from 1902 to 2009, for all natural and medical sciences journals. Basically this means that even if all you care about is the number of citations you get then even then the impact factors of journals is starting to matter less and less.

Of course, what matters is how influential your ideas are and not how much they are cited, but we don't really a have a good metric for that.

Answer 3

Clearly, people here don't seem to like the idea that impact factor would be used to evaluate journal quality. I'm not a big fan of impact factor either. It can be gamed (and the degree to which it is gamed will increase as greater importance is placed on it). It also becomes particularly problematic when you compare across disciplines with different citation practices and citation half-lives. Furthermore, there's also the risk that people start using impact factor to evaluate article quality, which is a lot more questionable.

That said, within a discipline, I generally find that better journals have higher impact factors. It's not perfect. But there is a strong correlation. There's also research that shows very high correlations between various indices of journal quality including peer ratings, impact factor, and various other citation based indicators.

So in short, within a field, impact factor is one of many variables that typically correlates highly with journal quality. If you know nothing about the quality of a journal, you'll know more about it by looking at its impact factor. However, That shouldn't stop you from looking at other and most likely better indicators of journal quality (e.g., what is your own evaluation of the content published in the journal in recent years).

It is also really important to ask yourself why you want to evaluate journal quality. Is it to select a journal? to reward, promote, or hire academics? Using impact factor in these cases can be problematic for a wide variety of reasons, but these are separate issues.

Here's one article that I found providing empirical evidence by Saha et al (2003). To quote the abstract:

Objectives: Impact factor, an index based on the frequency with which a journal's articles are cited in scientific publications, is a putative marker of journal quality. However, empiric studies on impact factor's validity as an indicator of quality are lacking. The authors assessed the validity of impact factor as a measure of quality for general medical journals by testing its association with journal quality as rated by clinical practitioners and researchers.

Methods: We surveyed physicians specializing in internal medicine in the United States, randomly sampled from the American Medical Association's Physician Masterfile (practitioner group, n = 113) and from a list of graduates from a national postdoctoral training program in clinical and health services research (research group, n = 151). Respondents rated the quality of nine general medical journals, and we assessed the correlation between these ratings and the journals' impact factors.

Results: The correlation between impact factor and physicians' ratings of journal quality was strong (r2 = 0.82, P = 0.001). The correlation was higher for the research group (r2 = 0.83, P = 0.001) than for the practitioner group (r2 = 0.62, P = 0.01).

Conclusions: Impact factor may be a reasonable indicator of quality for general medical journals.

References

• Saha, S., Saint, S. & Christakis, D.A. (2003). Impact factor: a valid measure of journal quality?. Journal of the Medical Library Association, 91, 42. PDF

Answer 4

"Quality" is very subjective. Impact factor is a good measure of how often articles in a journal are cited, but this doesn't necessarily directly relate to "quality".

Many journals with a narrow focus will have relatively low impact factors, but are still considered very "high quality" (i.e. prestigious to publish in) journals.

As an example, in the geosciences, the Journal of Structural Geology is a much more prestigious (and harder to publish in) journal than, say, Tectonophysics (just to pick another Elseviver journal), but it has a lower impact factor.

Answer 5

The editors of Epidemiology (a very good journal), recently had an editorial about this:

Most major epidemiology journals, including ours, have seen a steady rise in their impact factors during recent years. At the same time, the relative rank of these journals changes from year to year. Such changes are unlikely to represent true annual changes in these journals' relative quality. We think the various epidemiology journals are indeed different, and they deserve to be evaluated and compared. But we're happier when such assessments are based on matters of substance, such as editorial policies, quality of reviews, quality of editing, efficiency in the processing of manuscripts, and the (real) impact of the journal on the field.

Said editorial also has links to several others in that journal critiquing the notion of the Impact Factor, and its scientific merit (or lack thereof).