3

Measures of academic productivity, like the h-index, m-index, i10-index are useful but imperfect. Their imperfections are more than an abstract concern. Hiring committees consider them in determining how the applicant stands in relation to his peers. The weight committees give to these numbers varies, especially with newer metrics.

How do we know which measurement of scientific productivity is the most accurate? This question suggests that no measure is accepted as generally accurate. The h-index has a retrospective validation for some fields. Have studies tracked a cohort of scientists over time to compare the ability of these measurements to predict who received a tenure-track appointment in the next five years? (I realize that is not everyone's goal. It is one that tenure committees care about.)

Improve this question
7
  • 3
    If I had the answer, I would be publishing it in Nature (and collecting lots of citations, hehe). I find this question akin to ask what is the solution to Navier Stokes in Physics SE, or if P=NP in CS.
    – Davidmh
    Mar 7, 2015 at 1:10
  • 3
    Because promotion committees use these metrics in deciding who gets promoted, you cannot disambiguate causation.
    – RoboKaren
    Mar 7, 2015 at 2:07
  • 1
    With respect to predictive validity: ability to predict what? Citations? Tenure? Future h-index? Prizes? National Academy Membership?
    – Corvus
    Mar 7, 2015 at 2:26
  • @Corvus point taken, I meant tenure, edited.
    – mac389
    Mar 7, 2015 at 2:51
  • 4
    Not entirely sure if Goodhart's Law applies here. en.wikipedia.org/wiki/Goodhart%27s_law
    – chipbuster
    Mar 7, 2015 at 4:53

1 Answer 1

Reset to default
2

I don't know that anyone has done this on a broad scale for tenure. There is a fairly recent study that used a machine learning approach to figure out what metrics are predictive of success in achieving a faculty position.

van Dijk, D., Manor, O., & Carey, L. B. (2014). Publication metrics and success on the academic job market. Current Biology, 24(11), R516-R517.

For an overview and a simplied version of the application, see this Science Careers story.

From the paper's summary:

The number of applicants vastly outnumbers the available academic faculty positions. What makes a successful academic job market candidate is the subject of much current discussion [1–4] . Yet, so far there has been no quantitative analysis of who becomes a principal investigator (PI). We here use a machine-learning approach to predict who becomes a PI, based on data from over 25,000 scientists in PubMed. We show that success in academia is predictable. It depends on the number of publications, the impact factor (IF) of the journals in which those papers are published, and the number of papers that receive more citations than average for the journal in which they were published (citations/IF). However, both the scientist’s gender and the rank of their university are also of importance, suggesting that non-publication features play a statistically significant role in the academic hiring process. Our model (www.pipredictor.com) allows anyone to calculate their likelihood of becoming a PI.

tl;dr Publish lots of papers that get lots of citations in good journals. But you knew that already.

Improve this answer
3
  • 3
    This study, while a good descriptive work, does not separate cause and effect. It does not actually study productivity, but rather the way that people are judged on their productivity, which just brings us back to the original question.
    – jakebeal
    Mar 7, 2015 at 3:37
  • 1
    @jakebeal I agree one hundred percent. But in the post and my clarification comments, the OP specifies that he is interested in measuring productivity by whether or not people get tenure: "Have studies tracked a cohort of scientists over time to compare the ability of these measurements to predict who received a tenure-track appointment in the next five years? " This issue, like hiring, depends on the way that people are judged by their productivity. So I believe the study I describe is as close to what the OP seeks as one can find.
    – Corvus
    Mar 7, 2015 at 4:23
  • I wonder why gender is a factor. It suggests to me that the model is reflecting the biases of the data set. That's understandable, although the authors don't try to correct for that.
    – mac389
    Dec 2, 2019 at 21:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .