17

I have been reviewing for a particular prestigious journal in my field for some time. I noticed that the editor invites a significant number of reviewers (4-7) simultaneously. In most cases, not all reviewers submit their reviews in time, but in one particular case, 5 reviewers submitted a review, and the paper was rejected.

While I understand that it is common to invite a number reviewers because many end up declining or not submitting a review, I feel like it is unethical to do so simultaneously, and is a situation similar to an author submitting to multiple journals, as it "wastes" reviewer-hours. In the particular 5-reviewer case, the paper did not need 5 reviewers to be rejected.

Is this kind of practice common and/or is it considered ethical in general?

  • That does seem like a lot of reviewers but I would hate for reviews to take twice as long just so each reviewer could decline/accept in turn. – Azor Ahai Jan 3 '18 at 17:46
  • 13
    Why would it be "unethical"? An editor may ask for more reviews or fewer reviews, depending on the paper. Maybe he wants to get perspective from several different areas. If the paper is potentially very important (or very controversial) he would ask for many reviews immediately. – GEdgar Jan 3 '18 at 17:52
  • @GEdgar that's a very good point. – FBolst Jan 3 '18 at 17:57
  • As a possibly interesting datapoint: In fields such as CS where conferences are the prevalent venue of publishing peer-reviewed work, getting more than one review per submission (usually rather 3, possibly more) is the norm. Several requests for review are sent out simultaneously for each submission, and it is actually expected that all of them will be responded to roughly in time (granted, the process is slightly different as these reviewers had been asked before to signal their readiness to participate in the reviews). Nonetheless, it is very much expected that several experts spend ... – O. R. Mapper Jan 4 '18 at 11:39
  • ... time reviewing the same manuscript, indicating causing this situation as such is not considered unethical by any means. – O. R. Mapper Jan 4 '18 at 11:40
17

It's quite common to invite multiple reviewers simultaneously, though 4-7 is on the high side in my experience (yet sometimes, even 4-7 is not enough!). The numbers quoted in the details below are for physics, and might differ for other fields.

Regarding whether or not it's ethical, it really depends on tradeoffs. The tradeoff for not inviting multiple reviewers at the same time is that your paper will take longer to handle. It's not just submitting reviews on time. Not every reviewer invited will agree to review it! My personal guideline when I was an editor was that, each time I invited reviewers (3-5 at a time), I would budget one month till I receive reviews. That itself is an optimistic budget - 6 weeks is closer to the average, since there's a long tail in the distribution.

The alternative to inviting multiple reviewers at the same time is to invite one at a time, and wait till you hear back from the reviewer if (s)he is willing to review. This leads to the following problems:

  1. How long are you willing to wait to hear back from the reviewer? Let's say a week (which was also my personal guideline). Since you might have to invite 6+ reviewers before someone actually agrees to do it, you could easily be waiting 6 weeks before the paper starts being reviewed. If the reviewer takes one month to complete, that's at least 2.5 months before you can make a decision, not including any administrative time (i.e. you and the editor-in-chief both make decisions immediately).
  2. What if your journal requires two reviewers? You can invite two at a time, but there's no guarantee that both of them will agree to review. Maybe one will, the other won't. You could end up with one review submitted with the next one due in another month.
  3. What if the reviewer you invited writes a poor review (happens more often than one might think)? Sure you can blacklist the reviewer, but you still need to make a decision for the paper. If you invite more reviewers, again, that's another 1+ month till you receive the reviews.
  4. Nightmare scenario: what if your reviewer agrees to review but then does not actually submit a review, in spite of multiple chasers? If we take the worst case, you've already waited 2.5 months and then discover that you have to start from scratch, which can lead to another 2.5 months before you can make a decision. Ouch!

In the end it comes down to how long authors are willing to wait for their papers to be reviewed. If authors are willing to wait 6 months, then yes, editors can invite one reviewer at a time and waste fewer reviewer-hours. On the other hand it's common for authors to start worrying if they don't hear back in two months, in which case the editor really cannot invite one reviewer at a time.

  • 2
    Exactly as @user3727079 mentioned, the editor-in-chief recommendations / guidelines in many cases is to invite multiple potential reviewers. If many of them accept, then I do not see an issue with sending a follow-up e-mail to those who have not accepted and "Thank you but your review is no longer required. We are looking forward to your future contributions in our peer review....." and move on. – o4tlulz Jan 3 '18 at 20:59
  • Option #4 is probably made more likely when choosing to referee is "opt out" - if the reviewer does nothing, it looks a lot like they've accepted. (In physics, many Phys. Rev. journals seem to do this.) – AJK Jan 3 '18 at 21:31
  • One journal I regularly review for does as noted by @o4tlutz, they invite reviewers until 'enough' accept. If the first 2 reviews back make similar comments, the editor makes a decision quickly (same day) as to whether to wait for more. If not, the decision is made on those 2 reviews and the email out to the authors is copied to all who accepted, letting the other reviewers know that their review is no longer required. – JenB Jan 3 '18 at 23:01
  • 1
    @AJK If this is the case, the journal's interface is to blame. Pull your weight as editors and demand a better system; otherwise you are wasting the time of everyone in your field. – Federico Poloni Jan 4 '18 at 10:08
8

Editors are creatures of habit, just like the rest of us.

If they’ve been on the job a while, they generally have a good idea of how many invitations they need to send to get the desired number or reviewers. Sometimes, though, they end up with more than they expect. It happens. But I think the occasional example of this happening is better than inviting only a few reviewers at a time, and waiting for responses before continuing to the next group. That lengthens the time needed for review and slows down the cycle.

4

I've seen this in cases where referees were slow to respond, did not respond at all, or promised to write a review but never did. In the example below, that happened with anonymous referee #2. By the time the editor gives up on ever receiving a promised review, months may have passed.

many declined

If you give people a week to respond to an invitation, then three weeks to write a review, it may take a long time if many people do not respond to invitations quickly. Suppose 20% accepts a referee request and you want 2 reviews, then it makes sense to ask 10 people immediately, just to speed things up.

2

I've never seen this before so it doesn't seem common but an editor can solicit and pass forward as many reviews as he or she feels is necessary... Definitely not unethical.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.