27

Is the following common? An editor for a journal sends a paper out to, say, three or four referees. After getting the first two referee reports, the editor makes a decision and tells the third and fourth referees not to bother (so that they have potentially been wasting their time).

I can see this happening in some circumstances, e.g. if one referee finds a substantial error. However, I was asked to referee for one journal, and I get the impression (which I have not yet confirmed) that they typically send the paper out to lots of referees and take only the first couple of referee reports.

This practice, if it is indeed what they are doing, seems likely to produce quick turnaround times, but also seems disrespectful to referees who might be half-finished when they are told their reports are no longer needed.

Is this common? (In particular, in mathematics?)

  • There was a discussion of this practice in Sociology journals recently on the scatterplot blog. – Andy W Sep 3 '13 at 12:14
28

I think this one will be very hard to answer with hard data. So, I'll just throw in some anecdotal evidence, along with a few things learnt from some editors that I know.

From a reviewer point of view, it has happened to me exactly twice in a decade (i.e. very rarely) to receive an editor's review saying “I've reached a decision based on input from other reviewers, and you do not need to review the manuscript”.

  • The first time, the email came as my review was already overdue (a week or ten days), so I suppose the editor asked another reviewer when I didn't reply on time, and the new reviewer was fast to reply.
  • The second time, it was sort of the other way around: I was asked to review a paper, then 5 days later the editor wrote, saying the reviewer who was uncommunicative had finally replied, and my review was no longer needed. He apologized profusely, and offered to actually wait for my review if I had started doing it and wanted to finish. (I was happy to let it go.)

I think for an editor, growing a list of trustworthy and willing reviewers is crucial. So, they simply cannot make them work for nothing! (pun intended)

  • Let me point out that, as a referee, you could have been "overbooked" many times and never know it: if the editor does not write "I've already reached a decision" to you, there is no way for you to know that you have worked for nothing. – Federico Poloni Sep 2 '13 at 17:28
  • 5
    @FedericoPoloni I agree there is no way to know systematically, but in most cases articles for which modifications are advised/requested will come back for a second review, in which case you know whether you voice was heard – F'x Sep 2 '13 at 18:43
  • I'd guess that the first time, there was more than one reviewer invited, so the editor made a decision based on the reviews which had already been submitted. – Allure Feb 1 '18 at 23:27
20

An editor that cancels the review for a referee within the "normal" referee period will end up angering the referee and likely losing her as a reviewer for future papers. So this is a very unlikely practice.

Many journals will request multiple reviews (three is not uncommon for many of the journals I submit to), and will wait until the end of the standard review period to return any comments. If two of the three (or four) reviewers have returned reviews, and the others haven't, then the process can reasonably truncated at that point. (If you're behind schedule, you don't really have a right to complain in this case!)

  • 1
    To add support to your last point, I recently had an editor base their decision on only 2 of 3 reviews because the 3rd reviewer was quite late. – DQdlM Sep 2 '13 at 16:51
13

I know this question was asked and answered long ago, but I thought it might be useful to contribute some real data from an editor / program chair point of view.

Because reviewers are volunteer labor, they are unreliable. I personally find that I get about a 75% rate of return on useful reviews. For journals, pretty much every review that is actually returned is useful, but many do not return. For conferences, where the PC members have signed up in advance, the rate of return is much higher, but there are a significant number of essentially useless 1-sentence reviews that give a score but no justification.

At the same time, an important dirty secret of the reviewing process is that there is rarely a fixed number of reviews that are actually necessary. In the venues in which I am involved, typically 4 reviews is best practice, 3 is acceptable, and 2 is only supportable if there is very strong agreement on a clear accept or reject.

Moreover, when you are recruiting reviewers for a journal submission, the reviewers don't necessarily respond to the request to review immediately. Thus, I will typically significantly overbook the initial set of requests, asking 6-8 reviewers. Generally somewhere between 2 and 6 of those will accept, from which typically 3-4 will actually return reviews, giving me enough for a well-justified decision. If things go unusually well and end up with five reviews, that's just fine and will make the author feel we've taken them very seriously, but I'm not sure such an overabundance has ever actually happened to me.

If not enough reviewers accept, I have to send out additional batches of requests, all of which can lead to a significant skew in the times at which reviews arrive. Moreover, it is an extremely rare reviewer who will return a review significantly in advance of the due date. So for me at least, overbooking isn't to get quick turnaround time, but to prevent excessively slow turnaround time that can happen when you need one more review and a reviewer drops out at the last minute, forcing you to start the clock all over again on a new reviewer.

Because it is all volunteer labor, however (and because I do my own turns as a reviewer in other venues as well), I am very mindful of the importance of not taking a reviewer for granted. If somebody has promised me a review, I want that review. It is only when they have become significantly late that I will send a question along the lines of: "I am currently only waiting on your review before I can send a decision to these authors; are you still able to provide a review?" The only time when I simply cancel a review request is when the reviewer has become repeatedly unresponsive, and then they get a black mark for unreliability in the appropriate set of organizational records.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.