4

I was using Google Scholar to judge the rank of a researcher, but then realized I couldn't find everyone I wanted to and came across this question which highlighted this problem. Looking at Scorpus, which was suggested in that same answer, seems to downvalue any non-Elsevisor publications and not have any current data (from the last year). Microsoft Research seems altogether unintuitive to use and doesn't seem to have h-index at all. I tried PublishorPerish, but filtering data to find the correct researcher's papers seems ridiculously difficult.

I would like to be able to judge status somewhat programmatically, understanding there may be some flaws in this method, as it would be somewhat cumbersome to do it by hand on the scale I would like to. I will probably supplement it my sorting the top 3-5 by hand, but I would not like to do the entire set by hand.

Is there any proper solution to this problem? If not, what is the generally accepted method to do this?

At the behest of many people, I am a student, but this question is geared to generally understanding the current methods of researcher evaluation, not necessarily just by students

18
  • 8
    It'd probably be useful if you could explain the context more, i.e. why are you judging the researcher's status in the first place? In the context of choosing a research group it might be the wrong question to ask, for example. And the h-index has a large variability between fields and subfields, see e.g. this, so using it indiscriminately to compare two researchers' status is certainly overly simplistic.
    – Anyon
    Jul 13, 2020 at 19:50
  • 2
    @SolarMike No. I haven't posted and deleted a question recently. Jul 13, 2020 at 20:25
  • 4
    Yes, surely there are ways to compare researchers. Which way you choose depends on what your goals are, and for many of those goals you might not find any of the necessary information in a database. Search and tenure committees judge researchers all the time, but they don't rely on any one algorithm. Students judge them, too, in deciding who to work with.
    – Bryan Krause
    Jul 13, 2020 at 20:32
  • 9
    Do you want to judge status or quality? Many quality researchers may not reach the status they deserve because of their country, race, gender, or any other sort of biases. Jul 13, 2020 at 21:01
  • 2
    I would not dichotimize - it's not high vs. low, but rather I'm saying status is not a perfect predictor of quality, and is biased in known ways Jul 13, 2020 at 21:36

3 Answers 3

8

Assuming the question is about status and not quality:

Job title is a strong indicator of status. The meaning of each title typically depends on the country. For the US, as an example, named professorships have the highest status, followed by professors and associate professors.

The status of the university is also a strong predictor of status of the researcher.

"Status" is not quantitative and attempting to rank status is not productive.

4

The best signifiers of status in a community will be being invited to give (plenary/keynote) at the (top) conferences for that community, and to win the prizes and awards that community bestows.

Having many highly cited papers will often correlate with that (publishing papers other people are interested in contributes to gaining status, high status makes your papers more likely to be cited), but this can be gamed; and there will be cases where people earn high status with a single remarkable accomplishment not translating well into citations, stuff like that.

As mentioned in the comments, quality and status are different things. As much as we'd like academia to be a pure meritocracy, it isnt. In a similar line, you won't be able to see who gets invited to give plenary talks, only who accepts. Some people being unable to travel (ill health, no funding/remote location, care responsibilities) will distort the picture.

5
  • 5
    Conference invitations in some fields (the ones I am familiar with) are biased towards men. Jul 13, 2020 at 23:14
  • @AnonymousPhysicist I'd claim that that is a facet of women often not been given the status they deserve; not an issue with how well conference invitations work as indicator for status.
    – Arno
    Jul 14, 2020 at 8:07
  • 2
    My comment was context for your answer, not a critique. Jul 14, 2020 at 11:07
  • Your answer discusses post-invitation bias. I would upvote if you discussed invitation selection biases, per Anon's comments. Jul 15, 2020 at 20:58
  • Another issue here is the correct time-frame. Does someone with more invited talks in the past 20 years with no talks in the last five get ranked above someone with half the number of talks in the past 20 years but several talks in the past five? I think the question is impossible to answer without knowing for what purpose the ranking is to be used. Jul 16, 2020 at 18:14
-2

Use Scopus, because:

  • Sure it's maintained by Elsevier, but it also includes non-Elsevier papers.
  • Even if it does not have data from the last year, papers published in the last year probably have not have had time to accrue enough citations to change the h-index of well-established researchers.

Alternatively you could use a database such as Google Scholar or Web of Science.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .