In peer-review, is it common for a reviewer to be shown and asked to comment on other reviewers' reports?

Question

Some months ago, I was asked to review a paper for a computer science conference. I submitted my report about a week before the deadline, and I strongly recommended acceptance subject to a few minor corrections.

Just after the deadline, I was contacted by the programme committee member who originally asked me to review the paper. He sent me another reviewer's report (without a name or other identifying information) and asked for my opinion of it. The other reviewer had recommended rejection on the grounds that an example in the paper supposedly contradicted the claim of the main theorem. I explained that the other reviewer has misunderstood the example (in fairness, the authors of the paper could have stated it more carefully), and I see that the paper was ultimately accepted.

I have never before seen or been asked to comment on another reviewer's report, but my field is primarily mathematics, where there is often only a single reviewer for a paper, and I have only reviewed a few computer science papers.

So my question is this: in fields where it is usual to have multiple reviewers, is it common to be shown and asked to comment on other reviewers' reports?

A secondary question would be: if it is not uncommon, does it mainly occur when reviewers take opposing positions on a paper, like in the situation I describe?

Answer 1

I don't know about other fields, but this is a fairly common practice in computer science. Computer science has a very strong culture of significant publications in conferences, which have significantly different reviewing challenges than journal publications.

The key problem is that conferences typically have a large number of papers that all need to be dealt with by a fixed deadline and with a single iteration of revision. This generally means that:

1. You have to cast a wide net for reviewers, and are likely to be trying out a "new" reviewer on several papers at once, rather than iteratively.
2. People typically commit to reviewing far in advance, when the program committee is formed, and may find themselves with less time than expected and no opportunity to ask for an extra couple of weeks to produce a review.
3. There is no opportunity for the cycles of dialogue between reviewers and authors that happen during the journal manuscript revision process: generally there is no "major revision" option, just "accept with minor revisions" and "reject."

All of this adds up to getting a collection of reviews with more uneven quality and less chance to work the differences through in dialogue. Outlier reviews can be problematic: a very low outlier can cause an otherwise good paper to be rejected; a very good outlier can cause the authors of a rightfully rejected paper to feel that they were robbed of a publication.

One of the typical ways to try to deal with this (supported nicely by platforms like EasyChair) is to ask the reviewers to look at each others' reviews and to discuss differences. This way, when there are major differences of opinion, the more careful and informed participants can influence the others to adjust their opinions, just as happened in your case. Another mechanism (less commonly practiced, due to time constraints), is to allow the authors to write a rebuttal, which the reviewers are then asked to consider and see whether it leads them to adjust their scores.

There are other, more experimental methods as well---many computer science subfields take their conferences pretty seriously, and are constantly tinkering to try to improve them.

Answer 2

It's common for conferences that use the EasyChair system, for example, to show all reviews to all reviewers of a paper. Once you submit your review, you can see any of the others. With these kinds of conferences, the Program Committee (or a subset of it) often actually meets in some form, online or in person, to discuss all the papers under their scrutiny. This gives them an opportunity to figure out these kinds of problems. A good committee chair may ask one reviewer to revise their review in light of others if there was a mistake or misunderstanding.

That being said, I've never had this happen with journal reviewing. There, it seems more common for editors to take all the reviews into consideration and make these kinds of calls for themselves.

Answer 3

This isn't exactly the same situation, but in the first round of a paper I submitted in applied math, I received a glowing and a lukewarm/slightly negative review. When it came back to me, there was also what seemed like a tie-breaker opinion pretty much only saying "clearly worth publishing, but not in current form" (the paper was essentially copied and pasted from a thesis chapter which that review continued to say what it looked like). As the 3rd review made no substantial other comments, someone had clearly been asked to opine on the other 2 reviews, also underlined by, in later rounds, only getting feedback from the first two (obvious because of cross-references they made).

This gives anecdotal feedback to your question 2 - for journals, not conferences. It was a good journal in a rather dead field, in case that matters. I went to industry after and cannot provide insight into how common the above is.

Answer 4

It is fairly common in computer science for program committee members to have a large number of papers assigned to them for review. They will often review some subset of these papers themselves and send the others to sub-reviewers. (EasyChair makes it easy to do this and to keep track of everything.) Usually, the program committee members will see all the reviews, whether from other members or from sub-reviewers. The committee will discuss the reviews and try to iron out any disagreements. In some cases, but usually fairly few cases, this may involve going back to sub-reviewers, making them aware of disagreements, and asking for their help in resolving the issues. From your question, I infer that you were a sub-reviewer for one of the program committee members. In that case, being shown other reviews and asked to comment on them is not the most common situation but it happens for a small percentage of the papers in a typical conference (well, typical among the conferences that I have some experience with).

Answer 5

It is apparent that the standards vary between fields. In field with which I am familiar (in the Sciences) seeing and commenting on other reviews is not at all common (in fact I had not heard about the process before). Reviews typically go to an editor and is conveyed to the authors. Even in publications with open reviews where anyone can comment on the manuscript, there are specifically appointed reviewers that provide reviews that will be posted online as the manuscript is closed for further comments. this is the time when the authors get to respond and adjust their manuscript for continued consideration.

So the procedure you describe is not common to al fields by any means.