4

I have the impression that some reviewers have quality standards that they themselves cannot, or do not fully live up to when in the author role. However, since in most journals/conferences, the reviewers' identities are unknown to authors and fellow reviewers, it is difficult to check whether this impression is justified.

Some evidence

  • I have one colleague for whom I know that this is true. As reviewer, they often criticize papers for lacking supplementary experiments, e.g., ablation studies, sensitivity analyses, but as author, they often don't conduct such experiments (even when reasonable).
  • In peer reviewer discussions, I have witnessed that some fellow reviewers demand absurdly high quality standards, going far beyond what is typical (or possible) for the field.

Question: How common is this? I hope that someone who can see both sides, e.g., an editor or PC chair, might be in a position to give an answer based on facts.

Improve this question
9
  • 9
    Let's suppose it is like you suggest. What would be the conclusion? Would this have an impact how you handle their comments?
    – Snijderfrey
    Commented Jun 17 at 15:08
  • 3
    @Snijderfrey Well, I suppose if this is true for many reviewers, I guess I would take their comments less literally than I do now.
    – mto_19
    Commented Jun 17 at 16:02
  • 10
    That would be a mistake. Don't propagate others' mistakes, rather focus on improving the quality of your own work.
    – Snijderfrey
    Commented Jun 17 at 18:22
  • 13
    You can be an excellent baseball umpire without being able to hit the ball. Or a respected literary critic, etc.
    – A rural reader
    Commented Jun 18 at 0:33
  • 4
    FWIW, most of the papers I review have poor experimental evaluation that cuts corners in the interests of expedience. I try not to do that, which is probably why I publish so few papers. I am at the stage in my career where I only care about quality, early career researchers may be under more pressure to publish in quantity (e.g. for promotions), so while they would prefer to concentrate on quality, they have to deal with real world issues. That doesn't mean they are wrong to helpfully point out ways to improve the quality of a paper when reviewing.
    – Dikran Marsupial
    Commented Jun 18 at 16:38

3 Answers 3

Reset to default
17

This is not uncommon, but the root cause isn't that reviewers have high standards. As you've noted and I can confirm from my observations, these researchers often don't adhere to such high standards themselves. The reality is that they are busy people who lack the time to fully understand the papers they review. Instead of dedicating one or two days to thoroughly read, understand, and thoughtfully critique a manuscript, they make quick judgments. They skim through the paper and assess it based on its appearance, the interest level of the topic, the authors' affiliations, and their reputations. If a referee, using these superficial criteria, feels that a paper may not be suitable for publication in a particular journal, they look for formal reasons to justify its rejection. Citing the lack of ablation studies, sensitivity analyses, etc., is the easiest way to reach that end.

Improve this answer
6
  • 13
    This is a shallow generalization. This is more of an example of the worst of reviewing rather than the standard.
    – R1NaNo
    Commented Jun 18 at 14:28
  • 2
    @R1NaNo Unfortunately, this is not the worst of reviewing. The worst of reviewing is when the reviewer decides to write a negative review just because the manuscript is authored by a competitor. A mentor of mine explicitly asked me to help him find arguments to justify rejection of a manuscript written by a competing group. Compared to that, busy referees who write superficial reviews are angels :)
    – Mitsuko
    Commented Jun 18 at 20:22
  • 3
    @R1NaNo I also would like to add that in my experience, about 80% of referee reports are very shallow. Here's my post about one such report: academia.stackexchange.com/questions/186660 Unfortunately, most researchers are very busy and simply cannot afford spending enough time to properly review a manuscript.
    – Mitsuko
    Commented Jun 18 at 20:27
  • 2
    "I also would like to add that in my experience, about 80% of referee reports are very shallow." My experience as editor and associate editor (and also author) of various statistics journals is very different. I'd put the number closer to 20%. And I'd think I've seen around 1000 reports. Of course it may depend on the field.
    – Christian Hennig
    Commented Jun 19 at 8:48
  • 1
    @ChristianHennig Unfortunately, busy referees write their superficial reports so as to make them appear well-thought. It is only when you are the author that you fully see the actual quality of reports. Of course, as an editor, you must have seen a heap of reports, but many of them may be not as thorough as they appear to be.
    – Mitsuko
    Commented Jun 19 at 11:38
14

Looking at this from the point of view of an editor who handles a paper, the major important issue here is the quality of the resulting paper. If reviewers' comments make sense, it is irrelevant what these reviewers do in their own papers. That's just a different editor's job. If a reviewer asks for something that is important for raising the quality of the paper, this is a valuable request, and if the reviewer doesn't do that thing themselves when they publish, another editor and their reviewers should ask for it. They may not, but I have no influence on that.

On the other hand it is true that sometimes reviewers ask for things that may in principle have some value, but may require a huge effort for little effect, or would be better addressed in a different paper as the present paper is already long enough and only so much can be expected from a single publication. Once more, from my point of view, it isn't really relevant in such cases whether the reviewer would do such things in their own papers. Rather I'd expect the author to say in their reply to reviewers why a certain change/addition is not made. As an author I rather regularly do such a thing myself. Sometimes the editor is just fine with it (remember, it is not the reviewers who decide whether a paper is accepted, it's the editor). Sometimes the paper goes to the same reviewers again and it may happen that a reviewer is not happy that one of their issues has not been addressed. In this case they may insist and give arguments why they believe it's essential, and the editor may or may not agree. There may also be a path to compromise in that the reviewer still insists on something that was requested before but not all of it, and then the thing may go into another round.

The main point however is again that it doesn't matter for this what the reviewer does in their own publications. If you as author believe that certain requested changes are unreasonable, be it because you don't agree, be it because the effort would be unreasonable or it would go beyond the scope of the paper, you can defend your point of view in the reply to reviewers, and then it depends on whether and to what extent you can convince the editor and the reviewers - and it can happen for sure that an editor will accept something even if one of the reviewers (who may have asked for something unreasonable, and maybe for something they wouldn't do in their own papers, but that doesn't really make a difference) remains unhappy.

Regarding the more "empirical" side of the question, I've got to say that as editor I'm not usually interested, i.e., I will not check reviews against what the reviewers do in their own publications, because I think it's irrelevant to my job. However of course I often nominate reviewers whose work I know to some extent, and I rarely have seen strong discrepancies between reviewer demands and standard of their own work of the kind the question asks for (as a statistician, I do occasionally see reviewers asking for more simulations in methodological papers who try to get away with a minimum in their own, so it does exist to some extent).

Improve this answer
3
  • 3
    I think you highlighted an issue OP is talking about, the tendency of some reviewers to nitpick (i.e. requesting large changes for little payoff). Not sure if there’s a fix to that issue or if it’s just the cost of doing business though.
    – bob
    Commented Jun 18 at 16:12
  • 1
    I wonder if part of the issue is the perception that the job of the reviewer is to be a gatekeeper and decide whether a paper should be published, but their main job is actually to help the author(s) to improve their paper so that it is more effective in communicating the idea. Ablation studies, for example, often make a big difference to how convincing the experimental evaluation is.
    – Dikran Marsupial
    Commented Jun 18 at 16:35
  • 2
    @bob Remember that authors get to respond to the reviewer's comments. If you think something takes too much effort for too little payoff, include that in your response.
    – earthling
    Commented Jun 19 at 6:58
10

In addition to the other answers already provided, I think this also has to do with the role those people (or any of us, really) are in. As an author you have a different responsibility (i.e. assignment/task), and thus perspective (including drive and investment) than as a reviewer.

You will advocate differently as a result. Now would it be good if a bit of scientist/author understanding would be present in our review practices: Yes. Just as it would help if journals would give clear expectations and standards on what to do and not to do as a reviewer - and also if we all gave up striving for a Cell/Science/Nature paper and settle for "just" getting our data out there without all the pump and circumstance that inflates paper content.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .