Do editors ever "overbook" referees?

Question

Is the following common? An editor for a journal sends a paper out to, say, three or four referees. After getting the first two referee reports, the editor makes a decision and tells the third and fourth referees not to bother (so that they have potentially been wasting their time).

I can see this happening in some circumstances, e.g. if one referee finds a substantial error. However, I was asked to referee for one journal, and I get the impression (which I have not yet confirmed) that they typically send the paper out to lots of referees and take only the first couple of referee reports.

This practice, if it is indeed what they are doing, seems likely to produce quick turnaround times, but also seems disrespectful to referees who might be half-finished when they are told their reports are no longer needed.

Is this common? (In particular, in mathematics?)

Answer 1

I think this one will be very hard to answer with hard data. So, I'll just throw in some anecdotal evidence, along with a few things learnt from some editors that I know.

From a reviewer point of view, it has happened to me exactly twice in a decade (i.e. very rarely) to receive an editor's review saying “I've reached a decision based on input from other reviewers, and you do not need to review the manuscript”.

• The first time, the email came as my review was already overdue (a week or ten days), so I suppose the editor asked another reviewer when I didn't reply on time, and the new reviewer was fast to reply.
• The second time, it was sort of the other way around: I was asked to review a paper, then 5 days later the editor wrote, saying the reviewer who was uncommunicative had finally replied, and my review was no longer needed. He apologized profusely, and offered to actually wait for my review if I had started doing it and wanted to finish. (I was happy to let it go.)

I think for an editor, growing a list of trustworthy and willing reviewers is crucial. So, they simply cannot make them work for nothing! (pun intended)

Answer 2

An editor that cancels the review for a referee within the "normal" referee period will end up angering the referee and likely losing her as a reviewer for future papers. So this is a very unlikely practice.

Many journals will request multiple reviews (three is not uncommon for many of the journals I submit to), and will wait until the end of the standard review period to return any comments. If two of the three (or four) reviewers have returned reviews, and the others haven't, then the process can reasonably truncated at that point. (If you're behind schedule, you don't really have a right to complain in this case!)

Answer 3

I know this question was asked and answered long ago, but I thought it might be useful to contribute some real data from an editor / program chair point of view.

Because reviewers are volunteer labor, they are unreliable. I personally find that I get about a 75% rate of return on useful reviews. For journals, pretty much every review that is actually returned is useful, but many do not return. For conferences, where the PC members have signed up in advance, the rate of return is much higher, but there are a significant number of essentially useless 1-sentence reviews that give a score but no justification.

At the same time, an important dirty secret of the reviewing process is that there is rarely a fixed number of reviews that are actually necessary. In the venues in which I am involved, typically 4 reviews is best practice, 3 is acceptable, and 2 is only supportable if there is very strong agreement on a clear accept or reject.

Moreover, when you are recruiting reviewers for a journal submission, the reviewers don't necessarily respond to the request to review immediately. Thus, I will typically significantly overbook the initial set of requests, asking 6-8 reviewers. Generally somewhere between 2 and 6 of those will accept, from which typically 3-4 will actually return reviews, giving me enough for a well-justified decision. If things go unusually well and end up with five reviews, that's just fine and will make the author feel we've taken them very seriously, but I'm not sure such an overabundance has ever actually happened to me.

If not enough reviewers accept, I have to send out additional batches of requests, all of which can lead to a significant skew in the times at which reviews arrive. Moreover, it is an extremely rare reviewer who will return a review significantly in advance of the due date. So for me at least, overbooking isn't to get quick turnaround time, but to prevent excessively slow turnaround time that can happen when you need one more review and a reviewer drops out at the last minute, forcing you to start the clock all over again on a new reviewer.

Because it is all volunteer labor, however (and because I do my own turns as a reviewer in other venues as well), I am very mindful of the importance of not taking a reviewer for granted. If somebody has promised me a review, I want that review. It is only when they have become significantly late that I will send a question along the lines of: "I am currently only waiting on your review before I can send a decision to these authors; are you still able to provide a review?" The only time when I simply cancel a review request is when the reviewer has become repeatedly unresponsive, and then they get a black mark for unreliability in the appropriate set of organizational records.

Answer 4

Inviting more reviewers than is necessary is very common. In fact I'd be surprised if most editors didn't do this. The reason that you can't expect invited reviewers to always agree to review your article. Some of them will decline, some will wait a long time and then decline, and some will never respond.

Let's say your journal needs 2 reviews to make a decision, and you wait a week to hear back from your reviewer, and you start by inviting two reviewers. Suppose this happens:

1. You invite 2 reviewers (days past = 0)
2. After two days, one reviewer declines to say they have no time; the other doesn't respond. You invite one more reviewer. (days past = 2)
3. One week later, you invite one more reviewer, who also declines after a day saying they have no time. (days past = 8)
4. You invite two more reviewers, who don't respond. (days past = 15)

So it's two weeks after manuscript submission and the manuscript hasn't budged. You could argue that you have been respectful to your reviewers, but someone else could argue that you have been disrespectful to your authors.

What is not common is for journals to make a decision with reviews outstanding. Typically in this situation the journal will wait for the reviews, especially if it's only a short while till the outstanding reviews are due. Exceptions apply. For example, if after 30 days you have received two reviews, and then a reviewer who didn't respond suddenly accepts the invitation, with a review date 30 days into the future. In this scenario the journal could reasonably ask the reviewer to cancel.

Concerning this:

referees who might be half-finished

This is much less common than one might think. See source. The median amount of time spent per review is 5 hours. Unless the journal cancels exactly during these five hours, the reviews won't be half-finished; furthermore, if they do cancel exactly during these five hours and the referee says they are half-finished, the journal can probably wait an extra five hours before making a decision.

Granted, this might not apply to mathematics, where as far as I'm aware the time taken for review is measured in months, not hours.