How to evaluate researchers?

Question

Based on a recent question that seems closed or deleted, where the OP couldn't keep pace with his/her colleagues, I found it important to raise the questions of how researchers are evaluated to get promoted or appointed, etc.

• I observed two categories of junior researchers: 1) a lot of publications but mainly as co-author. 2) few publications but as the first author. If a junior researcher is assumed to demonstrate their capabilities in conducting research work and not only partially contributing to several papers, would the first category be considered more successful than the second one?
• The h-index and citation counts are basically out of the researcher's control and there are many ways to increase -falsify- them such as self-citation, etc. Are these metrics really accurate and why most of the decision-makers are still relying on them? Alternatively, what would be a good metric?
• If a researcher has N hours or days, what would be more beneficial for them, spending this time on publishing first-author papers or it is better to contribute to many papers and be a co-author? Here, my assumption is that the first authors spend significantly more time on the paper compared to the co-authors.

Answer 1

Where I work, everyone is evaluated according to some metrics. There are tons of metrics, from the number of articles, impact factors of the articles, citations, patents, grant money obtained, number of people trained, products (I'm not kidding - we are a research institution) and so on.

The evaluation itself is done by a secretary who just files away what we give her. There is a committee who is supposed to verify these. Those guys are simply out of their depth when it comes to what they are supposed to do, and it's not their fault.

This is not the norm in the western world, but I've seen similarly painful things done there, too.

The bottom line is that evaluating researchers is not straightforward for anyone. Surely, all kinds of metrics can be invented and they are relevant, until people start gaming them. And it is easy to game the metrics, as long as the people using them are the people financing the research and various managers. Even if one has decent advisers by their side, they can still make huge errors by looking at metrics.

The only people who can evaluate researchers are coworkers, collaborators and competitors. Everyone else might get dazzled by a recent Science paper, some other paper with 500 citations, or an extremely well prepared interview. Or they might dismiss someone because they don't have first author papers, or because they publish in less impactful journals or whatever.

This is one reason why, when hiring, many professors ask their prospective postdocs for recommendations. If someone they trust vouches for you, you get hired. It's even likelier to hire you if they collaborated with you on a project before and you had some success together.

Trying to read into what someone had done on his career path may give you some insight into how good they are at the job. But you need to look at hard facts. For instance, if someone has a few topology papers, you would expect them to be good at topology.

But, if you look at things like not having enough articles as first author, you can draw multiple conclusions: i) they don't like writing, ii) they just piggyback, iii) they lack motivation, initiative iv) they are lazy, but have some skill that is essential for their research group. In my case, it is because I second guess myself so much that I never reach the stage in which I'm happy with the paper. Any of those conclusions can be true, or none of them.

My point is you won't evaluate correctly a researcher until you understand their research and its context. Reading tea leaves (bibliometrics and other metrics) gives some information, but tends to obscure the truth. To give an example. I had more papers and an order of magnitude more citations than one of my office mates. We are in the same field. Yet, he got a postdoc fellowship at MIT, because the stuff he did was so advanced, my work looked like a science fair project in comparison. The place I work at now, would have dismissed my colleague without a second thought because "he's not a team player", "he publishes in low impact factor journals", "his work is not applied enough", "doesn't have enough citations", "he has a low potential to attract funding", and my favorite "his work is too technical".

Answer 2

The thing about evaluating people (for promotions, for raises, for annual feedback) is that every case is different. I've done that at two universities, for dozens of cases and the issue is consistently to figure out where the whole package someone presents falls. It isn't about the specifics of individual papers, and as a consequence, it's really not possible to give general feedback. It also wouldn't be practical in many cases: If you're a particle physicist, for example, you will simply not have the opportunity to write single-author papers.

My advice is generally to do what you enjoy and what you're good at, and you'll be ok. For some, like me, this means working in larger collaborations and figure out things together with others because I like the interaction across disciplines. Others are good at focusing on one specific topic for a long time, and they write single-author papers. Both categories of people would likely not enjoy -- and be less productive -- if forced to adopt the other strategy.