2

This is a quite broad question, to which I have some personal answers, but I think I need more different perspectives.

I am not talking about evaluating a peer's submitted work as a referee, but about evaluating applications ranging from small grants or sabbaticals (which have to be applied to at least in France) to tenure or tenure-track hiring. Such decisions have a small or huge impact on both others career and the overall academic system (by giving certain incentives), and should thus be taken very seriously. On the other hand, the time we are able to use for such evaluations is limited.

How should we deal with this opposing constraints (evaluation is serious matter but time is limited)? What tools and proxys should we use or avoid in the evaluation process?

I am not asking about how the system should be (I know my answer: more automatic funding, sabbaticals and pay raises to have less evaluations made more thoroughly), but about what one wanting to take evaluation seriously (as opposed to a formal game that only needs an arbitrary answer) can do in the current system. Actions that might imply change in the system are welcome, as far as they can be implemented individually and the plausible outcome is considered rather than the ideal outcome if everyone did the same thing.

Edit in view of comments

Clarification about "the system": if needed please precise in your answer the relevant bits of information about the evaluation system you are speaking about, as of course things evolve and vary from place to place -- but the broad principles should be applicable regardless of the fine details so please keep this as concise as is relevant.

Example of subquestions: it has often been said that impact factor, or more generally the glamor of journal titles should not be used for evaluation. Is there a consensus about this? I don't know. They are used in practice, but are there good alternatives? Are there specific evaluation circumstances when one should use this proxy?

What about h-index and other citation metrics? When should they and should they not be evaluated?

The question is not so much about how much time one should spend (which is more or less a given), but how to achieve a reasonable result in that time. Also, what to do in a situation where one feels a reasonable result cannot be achieved?

Improve this question
4
  • 4
    Is there any such thing as a current system? Each country has his own, which might be further constrained by law requirements.
    – Massimo Ortolano
    Dec 29, 2015 at 10:52
  • @MassimoOrtolano, and/or at least a current system at the local institution.
    – vonbrand
    Dec 29, 2015 at 13:01
  • Can you clarify what kind of answers you are expecting/hoping for? If the question really is "what can one wanting to take evaluation seriously do in the current system", the answer "one should take the available information (papers, promotion file, letters etc.), and spend as much time as is practical reading it and thinking about it, then reach a decision" seems pretty much like the unique, and rather trivial, answer. Maybe my imagination just isn't good enough to come up with something better? Anyway, it would be good to get a better sense of what kind of opinions you're hoping to solicit.
    – Dan Romik
    Dec 29, 2015 at 15:14
  • 1
    @vonbrand I've been at a fast growing engineering university for almost 15 years. The current system is quite ephemeral.
    – Fuhrmanator
    Dec 29, 2015 at 21:17

2 Answers 2

Reset to default
2

I'm only going to answer the first part of this question:

How should we deal with this opposing constraints (evaluation is serious matter but time is limited)?

It is true that it is often rather time consuming to do these evaluations. But I think ultimately, you need to apply the categorical imperative. In other words, you need to spend as much time and diligence on evaluations as you would expect your evaluators to spend on yours.

For a two-page application for a sabbatical, this may not be very much. But for someone applying for tenure, surely a thorough reading of at least the most important papers, along with some research about the impact of the candidate's work, is appropriate. This is thankless work, as you don't usually get a lot of credit for it. But it is nevertheless something that can't be taken lightly because of its outsize effect on the candidate and the importance of your evaluation. Consequently, if the case is not completely straightforward, if you find that you need to spend a couple of days on it, then that is what you need to do -- because you would your evaluators to do the same.

Improve this answer
0

This question is heavily opinion based, but I personally think that the system used in the British REF is pretty good: What are the three best papers the person has written in the last 5 years? Of course, the quality of a paper is also opinion based, but at least in mathematics you have measures like How many famous people have tried to achieve what this person does in this paper? Does the proof use methods which have previously been applied somewhere else? Does the result or the method have applications within or without mathematics?

Improve this answer
4
  • 1
    That is what is asked to the applicant, but what should the evaluator do with these three papers? Read them? Read the introduction? Look in which journals they where published?
    – Benoît Kloeckner
    Dec 29, 2015 at 13:33
  • 1
    Another point: I do not think this question is more opinion based than the large majority of other questions here; it may be more controversial, but that is another matter. I expect any opinion expressed as an answer to be backed by principle and facts, which should be easy in this case.
    – Benoît Kloeckner
    Dec 29, 2015 at 13:36
  • The evaluator does not have to understand the paper, but he should read enough to judge whether the problem was invented by the author or well known before, and see what reviews say about the paper. The name of the journal does not tell you too much about the quality of the paper. A paper in a top journal is never bad, but not always top. Vice versa top papers are rejected by all better than average journals, if their topic is far off the main stream. Also citation numbers say a lot more about which fields are popular, then which papers are important.
    – Jan-Christoph Schlage-Puchta
    Dec 29, 2015 at 16:22
  • 1
    your comment would be much more useful as an edit to your answer.
    – Benoît Kloeckner
    Dec 29, 2015 at 16:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .