As far as I understand, it is commonplace for editors to invite M reviewers when N < M would suffice. After N reports arrive, they take a decision and inform the authors. What happens to the N+1st, N+2nd, ... referee reports that arrive after the decision has been taken?

As a referee, I have only been told “thanks, we won’t need your report anymore” a very small number of times, and as an author I’ve never been told “look, there’s one more report for you”. So I wonder what happens in practice.

EDIT: I have in mind journals, not conferences. A conference is more driven by deadlines, probably.

  • 4
    as an author I’ve never been told “look, there’s one more report for you” – I have. – Wrzlprmft Jan 11 '18 at 12:51
  • In practice, the journals I where I know the process have a deadline for the reviewers to hand in. If they have N reviews at that deadline, those are the N used and the reviewers who hand in later might not have any effect. If they have less than N, they would send out reminders and extend the deadline. If they have M>N reviews at the deadline, all of them will be used in the decision and given to the author. – skymningen Jan 11 '18 at 12:54
  • @skymningen So the editor discards the surplus reports? Even when they suggest meaningful corrections that may be useful to the author? – Federico Poloni Jan 11 '18 at 13:01
  • 2
    @FedericoPoloni - the journals I'm familiar with would want the request to contact the authors of a paper you reviewed to go through the editor anyway. At the least that means that all parties are aware of the communication and that there is no 'back door' that could impact the integrity of the review process. – Jon Custer Jan 11 '18 at 14:20
  • 1
    I would really appreciate if we would get some first hand information from actual editors in the answers! – Dirk Jan 12 '18 at 12:01

From my experience, things happen differently. Comments do highlight it correctly.

Initially, N reviews are solicited by the editor. If they disagree (or, I have to assume, some reviewers miss their deadlines), more reviews are solicited. I've been reviewer #5 few times.

Typical values for N are:

  • 2, with third as a tie-break – a typical grant application review scheme;
  • 3, with 4 or even 5 as a tie-break - seen more often with journal papers.
| improve this answer | |
  • So you mean that typically N=M? There is no "referee overbooking"? So what is written here, for instance, is inaccurate? – Federico Poloni Jan 11 '18 at 22:54
  • In my experience (chair of a small-ish track at a conference, IPC member few times, reviewer since, like, 2008): there might be an "overbooking" in a sense that M>N, but if more reviews are requested than needed and they all arrive, all are regarded. So no "overbooking" in the sense "your review is not valid". – Oleg Lobachev Jan 12 '18 at 1:43
  • I think it's important to differentiate between a "request to review" and actual reviews. Sure, I'd ask more than N people to review. Then few decline, I'd still need to come up with at least N. If I get M>N it's not a problem. But all reviews that were submitted are considered. It's not like "oh, we got 10 reviews, so let's cherry-pick 4 out of there." – Oleg Lobachev Jan 12 '18 at 1:44
  • I have heard some editors only "overbook" during "tough times" when they admit that many reviewers are either very busy or on holiday. – skymningen Jan 12 '18 at 8:36

In my experience, typically the editor doesn't make a decision immediately after N reviews arrive. Instead (s)he waits for all the reviewers who've agreed to review to complete. This way the only chance of having surplus referee reports is if a reviewer agrees, does not submit by the review due date, editor makes a decision, and then the reviewer submits. This is rare, and gets even rarer the longer the editor waits after the due date.

If it does happen, and the original decision was 'revise', we write to the authors with "here's another review, please take that into account in your revision". If the original decision was 'accept' or 'reject', then the review is effectively wasted.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.