What is a fair metric for assessing the citation impact of journals across disciplines?

Question

To quote Thomsons "a journal's Impact factor is calculated by dividing the number of current year citations to the source items published in that journal during the previous two years."

I assume that disciplines vary in

• average number of citations per paper: Disciplines with fewer citations per paper will appear to have less impact.
• citation half-life: Longer half-lives means under-estimation of impact relative to journals with shorter half-lives. The Wikipedia article on impact factors summarises a study that found that "the percentage of total citations occurring in the first two years after publication varies highly among disciplines from 1-3 percent in the mathematical and physical sciences to 5-8 percent in the biological sciences." (Nierop, 2009).

Google Scholar uses the five year h-index. See this listing of top ranked journals with various psychology related keywords in their title. The five year h-index indicates the number of papers with an equivalent number of citations. E.g., a value of 20 indicates that 20 articles published in the last five years have received 20 or more citations.

However, while the h-index might reduce the issue of different citation half-lives, it does not resolve it. And it does not address the issue of differential citation patterns across disciplines.

Question

What index provides both a reliable and unbiased assessment of the citation based impact of a journal when comparisons are being performed across disciplines?

Reference

• Erjen van Nierop (2009). "Why do statistics journals have low impact factors?". Statistica Neerlandica 63 (1): 52–62. doi:10.1111/j.1467-9574.2008.00408.x.

Answer 1

There is no such index.

Publication and citation standards vary significantly between disciplines and even sub-disciplines. Without direct, deep knowledge of the standards in each community, it is simply impossible to compare impact of a journal in field X with the impact of a journal in field Y. (Eigenfactor's extraordinary claims to the contrary require extraordinary evidence, which they don't provide.)

Moreover, it's not clear why you should even try. Any judgement about the relative importance of Journal of X versus Journal of Y necessarily requires a prior judgement about the relative importance of field X versus field Y. HC SVNT DRACONES.

Answer 2

The CWTS SNIP factor controls for differing disciplinary citation rates and speed. See http://www.journalindicators.com It is also shown in Scopus

Answer 3

Here's an informal and difficult to quantify one: does it suffice to have a publication in this journal to get tenured? Or, in a reverse manner, can one get tenured without having published in a journal like this?

• In economics, you will get tenure in most reasonable places for publications in Quarterly Journal of Economics, American Economic Review, or Econometrica.
• In statistics, these would be Annals of Statistics, Annals of Applied Statistics, Journal of the Royal Statistical Society Series B Methodology, the Journal of the American Statistical Association, and Biometrika.

These are the fields I am closely familiar with. I can imagine that in most natural sciences, a publication in Nature or Science would boost your chances quite a bit. Finance has a system of ranking journals, with a handful of highly coveted journals designated as A-journals.

Getting reliable statistics along these lines is of course impossible. If Google were a little smarter, it would scan the CVs of those tenured and untenured (those who moved to another university after the typical period of 5-6-7 years), grab their publication records, and see what the journals are that those who got tenured had published in that untenured hadn't.

Answer 4

Moving from comments:

Defining field boundaries is an extremely difficult problem. Once those boundaries are defined (or as they are being defined) it is also difficult to judge which field individual journals belong to. The technical problem is one of "community-structure" or cluster detection, and is a big problem in computer science.

For the specific case of journals and citations, there is some cool work on this via statistical properties of random walks that makes nice definitions using the map-equation. However, these statistical methods sometimes produce oddities (I think I remember seeing Phys. Rev. Letters being classified as Chemistry based on the stats). It also doesn't always produce the fine-gaining desired, for instance eigenfactor seems to not have a Cognitive Science category, just a general Psychology category.

In general, I recommend taking a look around eigenfactor. Note that the website claims eigenfactor and article influence scores adjust for different fields, already (take a look at point #4). I am not sure how accurate this claim is, but I personally find their metric more reliable (not to mention more accessible!) than ISI Web of Science. I also think their approach is more developed than the freshly-pressed Google Scholar journal rankings based on h5-index. However, I would love to see eigenfactor and Google Scholar together.

If you are unsatisfied with eigenfactor's rankings, then a naive metric might be: classify journals into their fields and then assign each a ranking p, meaning the journal is in top p% of its field by some standard metric. This should give you a rough idea.