33

Is the following common? An editor for a journal sends a paper out to, say, three or four referees. After getting the first two referee reports, the editor makes a decision and tells the third and fourth referees not to bother (so that they have potentially been wasting their time).

I can see this happening in some circumstances, e.g. if one referee finds a substantial error. However, I was asked to referee for one journal, and I get the impression (which I have not yet confirmed) that they typically send the paper out to lots of referees and take only the first couple of referee reports.

This practice, if it is indeed what they are doing, seems likely to produce quick turnaround times, but also seems disrespectful to referees who might be half-finished when they are told their reports are no longer needed.

Is this common? (In particular, in mathematics?)

2
  • 2
    There was a discussion of this practice in Sociology journals recently on the scatterplot blog.
    – Andy W
    Commented Sep 3, 2013 at 12:14
  • I've heard this is common practice at MPDI publications. More generally, I think the former circumstance is more likely.
    – Zach H
    Commented Feb 3, 2023 at 15:14

4 Answers 4

32

I think this one will be very hard to answer with hard data. So, I'll just throw in some anecdotal evidence, along with a few things learnt from some editors that I know.

From a reviewer point of view, it has happened to me exactly twice in a decade (i.e. very rarely) to receive an editor's review saying “I've reached a decision based on input from other reviewers, and you do not need to review the manuscript”.

  • The first time, the email came as my review was already overdue (a week or ten days), so I suppose the editor asked another reviewer when I didn't reply on time, and the new reviewer was fast to reply.
  • The second time, it was sort of the other way around: I was asked to review a paper, then 5 days later the editor wrote, saying the reviewer who was uncommunicative had finally replied, and my review was no longer needed. He apologized profusely, and offered to actually wait for my review if I had started doing it and wanted to finish. (I was happy to let it go.)

I think for an editor, growing a list of trustworthy and willing reviewers is crucial. So, they simply cannot make them work for nothing! (pun intended)

3
  • Let me point out that, as a referee, you could have been "overbooked" many times and never know it: if the editor does not write "I've already reached a decision" to you, there is no way for you to know that you have worked for nothing. Commented Sep 2, 2013 at 17:28
  • 6
    @FedericoPoloni I agree there is no way to know systematically, but in most cases articles for which modifications are advised/requested will come back for a second review, in which case you know whether you voice was heard
    – F'x
    Commented Sep 2, 2013 at 18:43
  • I'd guess that the first time, there was more than one reviewer invited, so the editor made a decision based on the reviews which had already been submitted.
    – Allure
    Commented Feb 1, 2018 at 23:27
22

An editor that cancels the review for a referee within the "normal" referee period will end up angering the referee and likely losing her as a reviewer for future papers. So this is a very unlikely practice.

Many journals will request multiple reviews (three is not uncommon for many of the journals I submit to), and will wait until the end of the standard review period to return any comments. If two of the three (or four) reviewers have returned reviews, and the others haven't, then the process can reasonably truncated at that point. (If you're behind schedule, you don't really have a right to complain in this case!)

1
  • 2
    To add support to your last point, I recently had an editor base their decision on only 2 of 3 reviews because the 3rd reviewer was quite late.
    – DQdlM
    Commented Sep 2, 2013 at 16:51
21

I know this question was asked and answered long ago, but I thought it might be useful to contribute some real data from an editor / program chair point of view.

Because reviewers are volunteer labor, they are unreliable. I personally find that I get about a 75% rate of return on useful reviews. For journals, pretty much every review that is actually returned is useful, but many do not return. For conferences, where the PC members have signed up in advance, the rate of return is much higher, but there are a significant number of essentially useless 1-sentence reviews that give a score but no justification.

At the same time, an important dirty secret of the reviewing process is that there is rarely a fixed number of reviews that are actually necessary. In the venues in which I am involved, typically 4 reviews is best practice, 3 is acceptable, and 2 is only supportable if there is very strong agreement on a clear accept or reject.

Moreover, when you are recruiting reviewers for a journal submission, the reviewers don't necessarily respond to the request to review immediately. Thus, I will typically significantly overbook the initial set of requests, asking 6-8 reviewers. Generally somewhere between 2 and 6 of those will accept, from which typically 3-4 will actually return reviews, giving me enough for a well-justified decision. If things go unusually well and end up with five reviews, that's just fine and will make the author feel we've taken them very seriously, but I'm not sure such an overabundance has ever actually happened to me.

If not enough reviewers accept, I have to send out additional batches of requests, all of which can lead to a significant skew in the times at which reviews arrive. Moreover, it is an extremely rare reviewer who will return a review significantly in advance of the due date. So for me at least, overbooking isn't to get quick turnaround time, but to prevent excessively slow turnaround time that can happen when you need one more review and a reviewer drops out at the last minute, forcing you to start the clock all over again on a new reviewer.

Because it is all volunteer labor, however (and because I do my own turns as a reviewer in other venues as well), I am very mindful of the importance of not taking a reviewer for granted. If somebody has promised me a review, I want that review. It is only when they have become significantly late that I will send a question along the lines of: "I am currently only waiting on your review before I can send a decision to these authors; are you still able to provide a review?" The only time when I simply cancel a review request is when the reviewer has become repeatedly unresponsive, and then they get a black mark for unreliability in the appropriate set of organizational records.

1

Inviting more reviewers than is necessary is very common. In fact I'd be surprised if most editors didn't do this. The reason that you can't expect invited reviewers to always agree to review your article. Some of them will decline, some will wait a long time and then decline, and some will never respond.

Let's say your journal needs 2 reviews to make a decision, and you wait a week to hear back from your reviewer, and you start by inviting two reviewers. Suppose this happens:

  1. You invite 2 reviewers (days past = 0)
  2. After two days, one reviewer declines to say they have no time; the other doesn't respond. You invite one more reviewer. (days past = 2)
  3. One week later, you invite one more reviewer, who also declines after a day saying they have no time. (days past = 8)
  4. You invite two more reviewers, who don't respond. (days past = 15)

So it's two weeks after manuscript submission and the manuscript hasn't budged. You could argue that you have been respectful to your reviewers, but someone else could argue that you have been disrespectful to your authors.

What is not common is for journals to make a decision with reviews outstanding. Typically in this situation the journal will wait for the reviews, especially if it's only a short while till the outstanding reviews are due. Exceptions apply. For example, if after 30 days you have received two reviews, and then a reviewer who didn't respond suddenly accepts the invitation, with a review date 30 days into the future. In this scenario the journal could reasonably ask the reviewer to cancel.

Concerning this:

referees who might be half-finished

This is much less common than one might think. See source. The median amount of time spent per review is 5 hours. Unless the journal cancels exactly during these five hours, the reviews won't be half-finished; furthermore, if they do cancel exactly during these five hours and the referee says they are half-finished, the journal can probably wait an extra five hours before making a decision.

Granted, this might not apply to mathematics, where as far as I'm aware the time taken for review is measured in months, not hours.

1
  • My experience in reviewing for math is that the actual in-depth reviewing takes at least a couple of days (say 15 hrs total time). These hours are spent over the course of say a week, so most of the time is taken up by waiting for a suitable week (or to get to the front of the queue). However, if I have a 3-month deadline and an editor cancels after 2 months, it is pretty likely I have started to review. Commented Feb 3, 2023 at 9:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .