11

What would be the consequences, for instance, if the reviewer just rejected/accepted whatever paper they got, with a trivial or even nonsensical justification for it? Does the reviewer avoid doing that solely out of concern with the conference's quality?

7
  • 1
    In addition to possible negative consequences: there is little benefit to a reviewer from acting as you describe. At most, they might be able to remove one or two 'rivals' from the conference. This is unlikely to have any long-term impact on anyone's career.
    – avid
    Oct 18, 2021 at 13:11
  • 15
    Talking about these top computer science conferences, it is also important to note that while the reviewers are anonymous to the authors, they (often) aren't anonymous to the area chairs and senior area chairs who make the final decisions. Reviewers are also aware of other reviewers' full names in some conferences.
    – mirrormere
    Oct 18, 2021 at 13:57
  • 5
    When I've been a reviewer for a conference, every member of the program committee (usually a lot of people) could see all (usually 3) reviews for all the papers. So writing a dishonest review could really damage one's reputation. Oct 18, 2021 at 17:27
  • 1
    In a word, reciprocity. Oct 19, 2021 at 0:58
  • 1
    @AppliedAcademic, I read your remark as a sort of Golden Rule (to do unto others as you'd have them do to you), not as seems to have been (mis)read by EarlGray. Oct 19, 2021 at 21:10

7 Answers 7

18

I've been a program chair of an ACM conference before, and my take is that yes we sometimes do get reviewers who act this way. It's not a great look and it's a fraction of our total reviewer pool. Others have mentioned incentives, so here's what we've done (modelled off other conferences) with our bad reviewers:

  • Papers with one poorly-written review and two well-written reviews and that are borderline enough that we aren't sure (i.e. if the two well-written reviews disagree), we would tend to assign an extra reviewer.
  • There is a list prepared for the next PC team on all reviewers that wrote poorly-written reviews or for those who ghosted on their responsibilities, so they can be excluded.
  • Senior PC members and program chairs read these reviews. We're in the same field as these reviewers, so we now have a bad impression of their scholarly rigour and seriousness in exercising their responsibilities.

Some conferences have also started doing a "Best Reviewer" or recognition program for a handful of their most-dedicated reviewers. I think that's a good approach as some reviewers go above and beyond.

13

Positive incentives:

  • Building a positive reputation: This is especially relevant for early-career researchers (ECRs) who are not known by everybody in the field yet. ECRs that left a positive impression might be considered for future service roles and cooperative efforts. However, more senior researchers might still want to be perceived as being "good citizens" of the community.
  • Some conferences give out "best reviewer awards" - That is a nice token of recognition and CV entry.

Negative incentives:

  • Avoiding a negative reputation: Particularly poor reviews might lead to reputational damage, both for the reviewer and for the conference who let the poor review go through. Most people, to some extent, care about their reputation.
5
  • 5
    Hmmm. I once won such an award. And it was for rejecting a paper, actually.
    – Buffy
    Oct 18, 2021 at 13:07
  • 16
    @Buffy Rejecting a paper for the right reasons is a great service for the community! Oct 18, 2021 at 13:10
  • @Buffy You... Shall not... Pass!
    – Lodinn
    Oct 19, 2021 at 3:03
  • I just want to add that, in my experience, the reviewer (a senior researcher) usually delegates their assigned reviews to a junior researcher (usually PhD student) who will do the actual work. This way you have the incentive, building reputation, but without the downside of doing the actual work of writing a good review. Oct 19, 2021 at 11:49
  • @AlexbGoode That's indeed a common practice, but I wouldn't go as far as saying that it's the default. There might also still be some work for quality assurance required, as to avoid poorly written reviews being submitted, which would lead to reputation damage. Oct 19, 2021 at 11:56
9

They wouldn't be invited to participate in the future. Someone else would probably have to cover for them. But, beyond that, nothing is likely to happen other than a possible stain on their reputation. The program committee would see the results and they tend to be pretty connected in the field.

Reviewers tend, in my experience, to take reviewing quite seriously, both in an attempt to have a quality conference and to aid authors in improving papers.

7

There are no serious incentives here, people do this mostly for non-incentive based reasons. If you see yourself as a responsible person who cares about your profession then you’ll do reviews and try to do the best job you can under the circumstances. I’m not saying incentives never matter, but they’re not the only thing driving human behavior.

2

The main thing that prevents that is that reviewers usually self-select into the role. There are a few cases where conferences require you to act as a reviewer if you are an author of a paper, but aside from that, if you don't want to do a review, you just don't do it. Reviewers therefore tend to do reviews out of a desire for professional service more than anything else. It would be extremely unusual for an academic to elect to act as a reviewer but then (intentionally) give a vapid or nonsensical review. There is not any particularly effective incentive against this; it is just that there is no positive incentive to do it in the first place.

3
  • 1
    "extremely unsual" - that does not match with my experience. Even one-liner "reviews" of a manuscript submitted to a reputable journal indeed happen, and I would not say rarely. Why these reviewers volunteer to review is another question, to which I do not have an answer. I agree with your self-selection argument in general, though.
    – LuckyPal
    Oct 19, 2021 at 6:57
  • Due to the volume of submissions, many CS conferences now require all authors to agree to participate as reviewers when submitting a paper.
    – GoodDeeds
    Oct 19, 2021 at 10:54
  • @GoodDeeds: Thanks --- I've edited to reflect that.
    – Ben
    Oct 19, 2021 at 19:51
1

I think the question is wrong because it talks about "proper acceptance or rejection" as if proper were universally understood and agreed upon.

Instead of one universally agreed upon definition of proper, every conference can have its own understanding of proper and choose reviewers that mostly agree with the conference understanding.

You do not have to agree with the conference understanding, but if you disagree a lot then maybe you should choose a different conference.

1
  • Reading the whole question, OP seems to be asking about reviewers who don't make a genuine effort, not about reviewers who do make an effort but judge things differently to others. Oct 19, 2021 at 9:56
0

Nobody has mentioned yet that a single reviewer doesn't make the final decision. At least in top Computer Science conferences, each paper is reviewed independently by at least three different reviewers, then there is a discussion and the final decision is made by someone above them in the hierarchy ("metareviewer, etc.").

So not only would a reviewer look bad trying this, it wouldn't even work. They would be ignored or overruled by the other participants in the review process.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .