The impact factor (and other indicators of citations per document per year) is often used (controversially) to evaluate the research impact of a journal.
Critics of the impact factor often point out that it based on a skewed distribution with outliers. For example, many papers may have zero citations, another set may have a small number of citations, and finally a small set of papers may have a large number of citations (e.g., 50 or 100, etc.). In general, I've read that this profile is typical of many mid-tier journals. Thus, this is used to suggest that the impact factor may be influenced by outliers.
I agree that this distribution of citations does highlight that the impact factor overestimates the median citations per paper (as is the case with any positively skewed count distribution).
But another critique is that the impact factor is unreliable because of this distribution.
Thus, my question is:
What formula can be used to calculate standard error of the impact factor for a journal?
Presumably, this would help both critics and advocates of the impact factor to be able to quantify the uncertainty in estimating the impact factor. It would also assist in assessing whether it is appropriate to say that the differences in impact factor over time or between journals are a reliable difference.
- The standard error of the mean is commonly estimated as: s / sqrt(n), where s is the standard deviation of citations per article, and n is the number of citable articles. So presumably, the main factor that would reduce the standard error would be the number of citable articles published in the journal over the period. But unless we have the raw data, we wont have the standard deviation and the skewed distribution might lead this formula to underestimate the standard error.
- Use existing literature to determine the distribution of citations over articles. Hopefully, its a distribution that can be approximate by knowing only the mean, or where the mean provides a good estimate of the standard deviation
- Then apply a formula to get the standard error of the impact factor.