3

The impact factor (and other indicators of citations per document per year) is often used (controversially) to evaluate the research impact of a journal.

Critics of the impact factor often point out that it based on a skewed distribution with outliers. For example, many papers may have zero citations, another set may have a small number of citations, and finally a small set of papers may have a large number of citations (e.g., 50 or 100, etc.). In general, I've read that this profile is typical of many mid-tier journals. Thus, this is used to suggest that the impact factor may be influenced by outliers.

I agree that this distribution of citations does highlight that the impact factor overestimates the median citations per paper (as is the case with any positively skewed count distribution).

But another critique is that the impact factor is unreliable because of this distribution.

Thus, my question is:

What formula can be used to calculate standard error of the impact factor for a journal?

Presumably, this would help both critics and advocates of the impact factor to be able to quantify the uncertainty in estimating the impact factor. It would also assist in assessing whether it is appropriate to say that the differences in impact factor over time or between journals are a reliable difference.

Initial thoughts:

  • The standard error of the mean is commonly estimated as: s / sqrt(n), where s is the standard deviation of citations per article, and n is the number of citable articles. So presumably, the main factor that would reduce the standard error would be the number of citable articles published in the journal over the period. But unless we have the raw data, we wont have the standard deviation and the skewed distribution might lead this formula to underestimate the standard error.
  • Use existing literature to determine the distribution of citations over articles. Hopefully, its a distribution that can be approximate by knowing only the mean, or where the mean provides a good estimate of the standard deviation
  • Then apply a formula to get the standard error of the impact factor.
Improve this question
6
  • Why the down vote? Honestly, curious. Impact factors are relevant to academics. Presumably, being able to quantify the standard error of measurement would be beneficial both for people who like them and those that do not. For example, for those who are critical of impact factors, they could say "look, the difference between 1.7 and 2.2 is not reliable"; but that requires them to have some estimate of the standard error. Such information would presumably be useful in promotion, grant, and other contexts, where you are making an argument about research impact.
    – Jeromy Anglim
    Commented Jun 22, 2017 at 1:35
  • I wonder if journals provide their own raw data that will have a huge impact on their publicity policies.
    – Mithun
    Commented Jun 22, 2017 at 2:31
  • @Mithun you could probably get raw data from scopus or web of science fairly easily for one journal. You'd just need to work out how to do the correct dates for publication and citation windows.
    – Jeromy Anglim
    Commented Jun 22, 2017 at 2:57
  • Oh, I forgot. Another issue will arise as: when we try to obtain the citation count for an individual article within a time frame, how the citations outside this time will reflect the std? I know there is overall citations count and last five years citations count.
    – Mithun
    Commented Jun 22, 2017 at 3:29
  • 2
    @AlexanderWoo that is not true. Easiest counter example: the sampling distribution of regression coefficients in a linear regression model follows a t-distribution if the null hypothesis is true. More generally, the standard error is used to characterize the spread of the sampling distribution. You can use that for any sampling distribution. If you want to derive more from that (e.g. confidence intervals) you need to be careful to correctly apply the appropriate formula, but it is always good advise to be careful and do things correctly...
    – Maarten Buis
    Commented Jun 22, 2017 at 8:53

1 Answer 1

Reset to default
2

I think the standard error (of the mean, commonly estimated as s/sqrt(n) as stated in the OP) is a robust and good estimate for the uncertainty of the impact factor. It would certainly be interesting to know this together with the impact factors. Unfortunately, the standard error cannot be computed merely from the impact factor and the number of citable papers.

Note that the impact factor, being the average citation count, follows a normal distribution, even if the citation counts themselves don't (according to the central limit theorem).

A completely different question is whether, in view of skewed distributions of citation counts, the traditional impact factor (=average citation count) is a good measure for journal impact.

Improve this answer
1
  • So I guess that just leaves the question of whether the standard deviation can be estimated from the mean (i.e., the impact factor). My guess is that a reasonable approximation would be possible, especially if someone did some research. For example, my rough guess would be that the SD might be around 1 to 2 times the mean (but that the ratio probably gets smaller as the impact factor increases); but I'd need to see more data to know exactly how it works.
    – Jeromy Anglim
    Commented Jun 22, 2017 at 23:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .