How does sending manuscript to new reviewers close the review loop?

Question

I had sent a manuscript of mine to one of the IEEE Transactions journals a while ago. After the first round of revision, I received some comments of two reviewers (let's call them A and B). But after the second round of review, the new comments I received obviously indicated that the two new reviewers (say, C and D) were not A and B (both C and D had explicitly noted that in their comment letters). The comments of C and D were mostly contradictory to those of A's and B's. The weird thing was that both C and D complained about some changes which I had applied to the manuscript according to what A and B had requested. Yet, the most perplexing thing, about the most-recent comments I received after the next round of revision, that is, the editor apparently sent the new revision yet to a new pair of reviewers (here, E and F)! Again, their comments are no where close to something harmonious with what the former pairs had asked before.

Now, I am confused to understand the strategy of the editor. To me, a review process has to be closed just like a loop, in that those who comment on a manuscript should assess the resulting revision. Otherwise, anyone else would pose new sets of comments and ideas which may simply contradict the former ones.

Can one explain how this way of review moderation yields convergence to a decisive result?

Answer 1

Most probably what happened was the invited reviewers declined for whatever reason (too busy, on holiday, etc) and the editor is inviting a new set of reviewers because it is journal policy to only accept when they have two accept reviews (or they simply aren't confident enough to evaluate based on the original reviews). It is rare for this to happen four times, but it's not inconceivable.

Either way it's obviously not ideal for the authors. I'd suggest asking the editor for guidance. You can't outright tell them what to do, of course, but you could point out that the reviewers are contradicting one another and you are unsure what to make of it.

Answer 2

To me, a review process has to be closed just like a loop, in that those who comment on a manuscript should assess the resulting revision.

So you submit a first version of the paper. The editors invites some reviewers to submit reviews, and Reviewers A and B accept the invite. When Reviewer A accepts the invite, they have no way of knowing how the process will evolve. They know that they have to submit a review within a fixed amount of time, upon which three outcomes are possible:

• accept as is;
• reject completely;
• recommend to revise and resubmit.

The first two outcomes mean that the process ends there. But consider what happens when the third outcome happens. An amount of time X will pass for the editor to make a decision, and Reviewer A has no way of knowing how much time, particularly since Reviewer B may be late with their review. Then, the editor will give an amount of time Y to the authors to perform their revision. Maybe you, as author, are very eager, and you only use an amount Z<Y of that time. So at some point, you resubmit your revision, and Reviewer A will be invited again to give a review. But X+Z time has passed. When Reviewer A was invited for their initial review, Reviewer A had no way of knowing:

• whether any number of revisions would be necessary;
• how much time X+Z would be.

Maybe Reviewer A finds themselves overloaded with teaching tasks when the invitation for reviewing the revision lands in their inbox. Maybe they had set a month apart for redecorating their house. This is why reviewers cannot, in general, be asked to commit to reviewing more than one iteration of a paper at a time.

---

You have run into bad luck to have been confronted with a completely fresh slate of reviewers twice. It is not nice to have to chase after a moving target. The prudent thing to do is to summarize the conflicts in the feedbacks, and send a message to the editor. It is their responsibility to resolve such issues, but they may not be aware that any issue exists. So you should make the editor aware.

Answer 3

When reviewers contradict each other there shouldn’t be any additional problem because the algorithm is the same as with one reviewer: you either agree with someone’s position and then edit your manuscript accordingly, or you disagree and then explain why to the editors and reviewers. Either case it’s just positions to consider. It doesn’t matter if there are contradictory positions. Having 6 reviewers should strengthen your piece.