2

I have recently accepted to review for a machine learning conference. It's a conference with chairs, area chairs (AC), and then reviewers who are managed by the AC's.

This conference is double-blind, with a rebuttal phase, and what follows was never mentioned in the invitation to review, the review instructions, nor on the website of the conference. But regardless of this, I would like to understand what could be the rationale for it:

  • reviewers are assigned papers. Each paper is assigned to an AC, and reviewed by 3-4 different reviewers. Reviewers do not know the identity of the authors.
  • each reviewer sends their reviews
  • the authors get the reviews (do not know the reviewers' identities, nor the AC's), and write a rebuttal
  • the reviewers are given the rebuttal for the paper (still do not see the authors' identities), and see the other reviews. Moreover, they now also see the identities of the other reviewers for the paper. (The authors still of course do not know these.)
  • the reviewers (managed by the AC, whose identity is also known to the reviewers, discuss and update their reviews in light of the rebuttal and the other reviewers' reviews and comments.

Now, I do not understand the point of the part I emphasized, and it seems to me it breaks a key premise of peer review (even single-blind). Namely, now you have implicit and explicit pressure among reviewers.

What are the arguments for this? ("This" being: divulging the identities of the reviewers to the other reviewers of the same paper -- not only to the AC)

2

Historical reasons: Many conferences have switched to online PC discussions only recently, after having had physical PC meetings before. Since anonymity is not an option at physical PC meetings, a non-anonymous online discussion is closer to the experience of physical PC meetings.

Incentives for PC members. PC members do a lot of work without monetary compensation. There are some reputation-related incentives they may have from a non-anonymous online discussion: Junior PC members can gain reputation, and senior PC members can prove they are still good citizens of their community. Likewise, PC members can get an impression of whom to invite and not to invite as reviewers for their own conferences and journals.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.