When should we use publication list for peer evaluation?

Question

This is a second attempt to ask about peer evaluation, after a pretty unsuccessful one which was probably much too broad. Let me start again, with a more precise subquestion.

The point is commonly made that one should not judge one's peer by looking at the impact factor of journals where her papers are published, as IF is a very poor indicator of the citation rate of individual articles, and certainly even poorer at asserting the intrinsic merit of individual articles. Some go further and consider that one should not judge one's peer by looking at the prestige of journals her papers are published in (e.g. because more selective journals tend to publish less reliable science). Last, the sheer number of publications is obviously no more useful to judge one's peer than the sheer number of books written is useful to judge a writer's merit. So, following these principles, one is lead not to use publication lists in peer evaluation.

However, publication list is most certainly the main component of a record in many, if not most, evaluation.

I have several questions on this paradox, to which I have personal answers which are partial and non definitive.

Q0 What are the actual practices you witness about the use of publication lists in peer evaluation?

Q1 Are there circumstances where using IF, journal prestige or other aspect of publication list leads to a better evaluation than not?

Q2 What other proxy if any can be used for evaluation that are expected to be done rather quickly (e.g. extracting a short list from dozens to hundreds of applicants for a tenured position, attributing small fundings), and what are their pros and cons?

Q3 What criteria can be used for evaluation that are expected to be done more thoroughly (e.g. hiring for a tenured position once a short list has been established), and what are their pros and cons?

Answer 1

Q0. Short answer, all kinds; from trying to dig into all details you mention in your backgroudn text to simply looking at the number. My experience says that evaluations often have lofty aspirations but when it comes down to actually performing the evaluation, a time crunch or other circumstances makes evaluations boil down to simple statistical measures.

Q2. The short answer here is, we do not know. In order to say that one approach yields better results than any other we need some form of comparable materials. I have not seen such comparisons made (which of course does not mean they do not exist).

Bibliometricians work with more advanced tools to try to work out comparable numbers. Having been through a few such processes, for organisations not persons, I can say that the results often indicate whether the object is good, average or perhaps poor and not much more; very relative terms. In my experienced studies the litterature is dealt with after having been subjected to, for example, field normalization since different fields have very different publication strategies. This i sone reason why a straight IF or h-index has limited value; it is important to know with what one is compared.

Q2. What immediately comes to mind is funding. If one is looking for a person that will contribute to a department through their activity, looking at their publications may not reveal the whole. Ability to attract funding may be equally important. It is obvious that the two may be well correlated in many cases and really poorly in others so even this would not be straight forward. Again, one has to look at the norms for the particular field to see how these can be used.

Q3. Thhis goes back to my answer under Q2 to some extent. I think, however, that in order to successsfully select a person for a tenure track, one has to be very clear of what one expects from such a person. Looking at publication rate is one thing but let's say that person locks themselves into a room and is never seen except whena new paper is published. Is that what the department really was looking for?

Appointing (good) people (contributors) is a difficult task and I very much doubt that using simple tools is the best way. Surely some tools such as IF end the like can be used to narrow down a larger group but the smaller the group from which to select, the more obtuse such tools become. I believe multiproinged approachs where the search criteria are clearly identified from the beginning and followed up by an assortment of means is the way to go. This usually involves interviews etc. to figure out how a person can and will integrate and contribute to the work environment.

So the short of it is that you get what you deserve if you use simple tools and underestimate the effort that is needed in the recruitment process.

This answer is very relative, I know, but I am convinced that hiring people is so much more involved than what simple statistics can resolve.

Answer 2

It is true that looking at a measure of the quality of the journals a colleague publishes in (such as your personal or your community's assessment of a journal or, if need be, the Impact Factor) is, by itself, a relatively poor indicator. It's also true that the number of journal articles is, by itself, a poor indicator.

But taken together, these two are certainly highly indicative: If someone has half a dozen publications in Science and Nature over the last decade, then she's clearly doing good science. If someone has 50 publications over the last decade but all of them in journals you've never heard of and published in second and third world countries, then that also tells you a great deal about the candidate.

You can make similar arguments about most other combinations of criteria that have to do with bibliometrics: taken in isolation, they don't tell very much; but if you consider two or three of them together, they are highly correlated with the quality of science someone does.

(I will add the following, because this argument has been made so many times and I'm tired of it: of course, the fact that there is a strong correlation between a combination of metrics and the quality of science neither implies that (i) there are no examples of candidates for whom the metrics incorrectly predict poor science, (ii) there are no examples of candidates for whom the metrics incorrectly predict great science. An example for the first point above may be Andrew Wiles pre-FLT, who did not publish much. But the important point is that just because there are outliers does not mean that the approach has no merit. You will, in most cases, be easily able to spot the outliers once you use the combined metrics on a candidate. It just requires a human inspection. But the inspection requires less work, and the results are far easier to document by taking into account metrics.

The argument that no metric, in isolation or combination is reliable is, in my view and observation primarily made by people whose metrics do not look great (but reasonable), and whose overall science career does not look great (but reasonable). They would not have much to win if someone took a quantitative approach to their annual evaluation. But just because a sizable number of people claims that metrics are "plainly unusable" to evaluate candidates does not make it so.)

Answer 3

I find that a publication list plus citations lets me answer three key questions about a person:

1. Has this person done a least some things that people care about a lot? (i.e.., a few highly-cited papers)
2. Is their work generally respected by their community? (i.e., many moderately cited papers)?
3. Has this person been consistently active in recent years? (i.e., good numbers of recent publications)

The numerical thresholds and definitions for judging all of these depend, of course, on field and career stage. I do not however, find publication venue to be a particularly interesting or convincing metric: strong publications in good venues are typically well-rewarded with citations, and I find that includes them in my evaluation well enough. To also count venue would effectively be double-counting.

Note that there are also key things that do not show up effectively on publications lists. For example, a person might have made a major contribution in software or community organization that would show up elsewhere on a CV.