26

Whether we like it or not, modern academia is increasingly being measured, in some vain attempt to get objective measures. Although it is unwise to fight 'being measured', it is at least possible to steer the measures away from meaningless ones, backed by peer-reviewed research that establishes this unrefutably.

There are a lot of different metrics that have been defined - I am not looking for those. What I want is pointers to the research behind the scientific validity of those metrics.

So the question is: where should I look for scientific assessments of bibliometrics?

Improve this question
2
  • 1
    Could you clarify your question? Are you asking for a listing and definition of the "scientific productivity" indices? Or, are you asking for a comparison and to start a discussion on the development of another one? Mike
    – mac389
    Feb 14, 2012 at 20:46
  • 1
    I do not understand what you look for: there are easy-to-compute measures that are measurably well-correlated with scientific productivity. Once, one starts to use any of them for making career-impacting decisions, the people whose careers are affected respond to the stimulus. There is no way one can have an irrefutable meaningful statistic.
    – Boris Bukh
    Apr 13, 2012 at 20:11

4 Answers 4

Reset to default
11

Johan Bollen and Herbert van de Sompels are two researchers to follow in this area. Bollen did an analysis of 39 different citation-based metrics which is a good place to start. However, it's crucial to note that there are serious errors in trying to use citation-counting methods as some sort of ground truth. Citation counting is problematic because:

  • Different fields have different citation practices. In biology it's common to have 10 or more authors on one paper, whereas in math you often have only one or two.
  • Citations take a long time to accumulate, penalizing early-stage researchers.
  • Citations only tell part of the story, leaving out the useful contributions made by researchers in the form of code written and datasets released.
  • Citations often mutate over time.

It's now possible to get more information about a paper than just who cited it, and it's possible to get this information before several years have passed and before the information about the impact of the paper becomes old and less useful. The Public Library of Science makes detailed article-level metrics available and Mendeley has an API from which you can collect real-time data about how many readers a paper has, as well as social metadata such as tags and annotations and reader demographics. These metrics are being consumed by services such as Total Impact and combined with data from Github, Twitter, and traditional citation metrics. My bet is that if you're looking for a meaningful set of measures, you're going to find it in these richer sets of aggregated data.

Improve this answer
0
19

Although just a start, we had a "journal club" over at the stats.se site on such bibliometrics, and had a chat over this particular article;

Arnold, Douglas N. & Kristine K. Fowler. 2011. Nefarious Numbers. Notices of the AMS 58(3): 434-437. PDF link from publisher

Abstract from initial ArXiv print:

We investigate the journal impact factor, focusing on the applied mathematics category. We discuss impact factor manipulation and demonstrate that the impact factor gives an inaccurate view of journal quality, which is poorly correlated with expert opinion.

As far as scientific assessments the authors in the above article are pretty negative of such rankings, and give a few examples of editors having citation practices that intentionally inflate their journals rankings.

I'm sure more literature on the topic exists than this though (so I look forward to any other suggestions).

Unfortunately the transcript from the chat is currently not linked to in the applicable thread on meta stats. But I will update here if it becomes available.

Improve this answer
1
  • 1
    This has been published in the March 2011 Notices of the AMS, vol. 58, no. 3, pp. 434-437. ams.org/notices/201103/rtx110300434p.pdf. The paper lists additional references that could be of interest.
    – JRN
    Feb 14, 2012 at 22:31
4

Concenrning mathematics as in the answer of Andy W, the International Mathematical Union has a report on possible policies to measure and rank journals.

Improve this answer
2

I'm against "productivity-based measures" like the h-index, to the point that I wrote an article on an alternative measure. This measure is based on impact, and it tries to remove productivity as a factor for evaluating scientific work. The article is open, and you can find it in Scientific Reports.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .