Assigned to Review a Paper I Previously Reviewed

Question

In a previous conference I was assigned a paper, which I rejected and the authors later withdrew with no comments. Now, in another conference, I was assigned the exact same paper. Choices:

1. Refuse to review it
2. Inform the Area Chair, proceed as normal, and append the previous review
3. Inform the Area Chair and then proceed as normal without mentioning previous review
4. Not mention my previous review at all to anyone

What would be the most ethically appropriate action?

Field: Machine Learning & Computer Vision, if it's relevant

Answer 1

If the authors did not change their paper in response to your review (which I assume you mean by "I was assigned the exact same paper"), then:

1. Your review is still as relevant as it was when you first wrote and submitted it
2. The authors wasted the time you spent reviewing

Here is another point to keep in mind, which reinforces point 1 above:

3. You seem to be well-regarded in this community, since you got the same paper to review from two different conferences

In light of points 1 and 3 above, I would simply send in the exact same review you already submitted earlier. This way, you provide value at no cost to yourself.

In light of point 2 above, I personally would certainly call out the authors on this. Wasting your time by not even editing the paper in response to your comments is, in my opinion, contrary to the spirit of peer review and of the entire scientific publishing workflow. The authors should be made aware of this, and so should, in my opinion, be the Area Chair.

However, it is possible that your original review did not reach the authors - issues happen (although my subjective expectation is that it is far more likely they did get the review and simply decided to ignore it). So I would not cast my comments as "the authors wasted my time", but rather as "the authors seem not to have received my earlier review, which I am appending here, since it therefore is still as relevant as a while ago". Both the authors and the Area Chair will get the message.

Answer 2

This is becoming incredibly common in Machine Learning and Computer Vision. Too many papers, too much interest in the area, not enough good reviewers, and no good second-tier or third-tier conferences that would act as a relief valve. The result is that the same potentially poor papers are getting resubmitted verbatim over and over again, potentially many times, in the hopes that they will eventually be stochastically accepted.

See the 2014 NeurIPS experiment, where the program committee sent some of their submitted papers through two separate reviews. The goal was to find out whether the accept/reject recommendations would be consistent. The greatly simplified explanation of their result is that really good papers will always be accepted, and really bad papers will always be rejected, but for the large bulk of the papers of middle quality that are rejected, there's a good chance your paper will be accepted if you just resubmit verbatim next time. Many authors have interpreted this as meaning that a rejected paper has about a 25% chance of being accepted if you just resubmit next time.

This is a problem that AI/ML/CV is dealing with right now. There's not much you can do about it. If the paper is unchanged, then copy/paste your review from last time and move on.

Answer 3

I certainly do not think it would be unethical to review it but my inclination, mostly in the interests of transparency, would be to tell the Area Chair and ask them to assign it to a different person. They might do as you ask, or make their own decision to reject it on the basis that they trust your viewpoint and do not see the need to waste the time of another reviewer.

Answer 4

I had this exact same situation happen to me (also in the same field, ML/CV): I reviewed a paper at conference A, recommended rejection, the authors withdrew with no rebuttal, and I was again assigned the same paper (which was in a near identical form) at a subsequent conference B.

What I did was write a private comment to the area chair informing them that I had already reviewed the paper and that the version I was being asked to review again was nearly identical. I was advised that submitting a similar review again was fine, as long as it reflected the current state of the paper, taking into account any changes that had been made between the two versions.

So in the end, I did something along the lines of option 2, without explicitly mentioning to anyone else (i.e. other reviewers, authors) that I was reviewing the paper for a second time.