How can I tell if I'm being too picky as a referee?

Question

I am a mathematician, frequently asked to referee papers. (As my career progresses, I now find myself frequently asked to referee good papers.)

I have found that I've gotten pickier and pickier as a referee. I just now finished a referee report (for an excellent paper, submitted to an excellent journal) with 53 bullet points on it: mistakes I found, requests for clarification, other suggestions.

On another occasion I believe I submitted six "revise and resubmit" reports for the same paper, before finally recommending acceptance.

In all these cases I am spending a lot of time reading the papers (which is worthwhile; they're interesting papers!), and I'm almost as meticulous as if it were my name on the paper.

How can I tell if I am going overboard with this? I have never heard any negative comments by anyone – including by journal editors, whom I asked for feedback on this matter after sending my reports. Indeed, editors have acted extremely happy that I've read these papers in such close detail. Nevertheless, I wonder if I am investing too much time in this, and/or annoying the authors.

Answer 1

In my humble opinion, it looks like you are the ideal reviewer actually! You give a lot of advice to improve the paper, and this directly benefits the authors and the journal. And apparently you give very precise advice, which is much more useful and actionable than general or vague remarks.

My main concern as a reviewer is to be fair in my final recommendation. As long as your meticulousness doesn't lead you to reject potentially good papers, you are doing a good job as a reviewer. However the question of whether you are spending too much time on it depends on your priorities, it's important to weigh the benefits and costs for yourself before you accept. It's perfectly acceptable to refuse a review from time to time in order to maintain the level of quality for the ones you accept.

Answer 2

It sounds like you are doing fine. It is in everyone's best interest to have high quality work and presentation. The authors don't need to take every suggestion you make, but are wise to consider what you say in each case.

But if you are overboard, you will hear from the editor. As long as you keep getting papers to review, don't worry about being too hard. Feedback is good for everyone.

Answer 3

If you manage to finish a report within a few months, then a careful detailed report is great (and it will make the author happy to see that at least one person really read the article). Having to choose between a superficial report within a month and an extensive list of all typographic and stylistic issues 2 years after submission, I would still prefer the superficial one.

And: Some things are a matter of personal taste. It would be nice not to request an author to rewrite a paper using different notation or completely change the structure or presentation, just because you (and maybe 60% of the people in the field) would prefer it that way (as long as it is still reasonable and not completely uncommon to do it the author's way).

Answer 4

Virtually every author is happy if others are reading their papers in detail. Neither is the editor going to object - highly detailed reviews are great from their perspective too. So you won't be going overboard on that front.

If anyone is unhappy you are being "too picky", it'll be on your end. Maybe you spend so much time reading papers that your PhD students / your own projects are being neglected, for example. Therefore you'll know the answer to this question better than anyone else. As long as you don't need the time you spend reviewing papers elsewhere, it's all good.

Answer 5

How can I tell if I am going overboard with this? I have never heard any negative comments by anyone -- including by journal editors, whom I asked for feedback on this matter after sending my reports.

I would consider the following indicators of being too picky:

• You make extensive remarks on language that the copy editor can address, i.e., subject knowledge is not required to spot and correct them. Exceptions are persistent mistakes such as a complete lack of articles or some examples to illustrate that the quality of English of the paper is not tolerable.

• You persist on opinion-based matters (other than the relevance of the work), such as notational paradigms, structure of the manuscript, or level of verbosity. By opinion-based I mean that somebody else might consider your suggestion detrimental. By persist, I mean that you keep picking on such an aspect, even though the authors clearly have an opposing preference (usually expressed in a reply to the reviewers). While you should mention most such aspects the first time you are seeing the respective material, the review-process should not become a back and forth about such this.

• You effectively end up doing the authors’ job and write the manuscript for them.

Apart from this, I do not think that a review can be too detailed. At the end of the day, you are giving recommendations to the editor and authors and it is their job to responsibly make use of them. However, it may be wise to take a few precautions to prevent irresponsible use:

I have never heard any negative comments by anyone -- including by journal editors, whom I asked for feedback on this matter after sending my reports.

I would not read too much into that fact. In most fields if not all fields, reviewers have become a scarce commodity and editors will avoid disgruntling them at almost any cost. If you want to make sure that your recommendations and review are not misinterpreted in terms of severity, or the categories offered for the overall recommendation are not nuanced enough, write a small note to the editor that elaborates your overall judgement and in particular how severe you consider the flaws you commented on. This also avoids that an editor makes a false blind decision due to not reading your entire review.

It may also make sense to consider that the authors may be overly obedient to your recommendations. If some of your suggestions are just this, make this very clear. For example, if you think that the work presented in the manuscript may be relevant for some application but are not sure about this (which is fine, since it’s the authors’ job to find out), make it very clear that you do not think that the authors should write this but just that they should consider this.

On another occasion I believe I submitted six "revise and resubmit" reports for the same paper, before finally recommending acceptance.

It’s hard to judge this without knowing why this happened. For example:

• In the first round you requested some additional proof (or other substantial addition of material) because the paper was incomplete. The authors added this, but the new material has a gap of roughly the same severity as the original gap. This processes then repeated until all gaps were filled. Assuming correct assessments from your side, this is probably not your fault but the authors’, since they failed to properly write a paper or to do rigorous research.

• If the six revisions were only about how to best phrase some sentence, you have been overdoing it.

Answer 6

There are 2 separate issues:

• The time you invest into finding things you think could be improved.

• What you threshold for withholding recommendation.

For the first:

More work on your part to identify areas for improvement, or that will make the fix for the author more clear, will invariably be welcome. Anyone going for a prestigious publication will have already spent significant time going over that sort of thing. An outsider engaging in that process constructively is help. Its work they would otherwise be doing, likely less efficiently as an outside perspective is useful.

The second is less clear:

Balancing quality of output vs time to publication is clearly a non-trivial compromise. Either extremes have obvious issues. I feel ill positioned to advise you but there are some things to be careful of:

There is (just?) fear in some communities that upsetting reviewers will hurt their careers. Hence being polite even in the face of obstinate behaviour is common. Hence I think there is a real danger that some behaviour (not necessarily yours) would cause ill will, or at least not be productive, with no obvious warning signs. This is even more true if there is suspicion that withholding approval is a political move (but this is pretty field dependent).

This is partly out of your control. However being clear about your expectations and having them distinguished from thoughts for improvements will improve the situation.

Is it worth it? Should you do less?:

Hard to say. I think it's important, and is seems so do you, but my judgement doesn't matter.