10

I was reviewing a paper for a conference. As I wrote in the review, it was IMO a completely average paper (boring problem, boring techniques, average write-up). I gave a scoring of "0" (ranging from -3 to +3) noting that it could, potentially, deserve publication if there is any space.

After submitting my review, I received a message from the program chair saying roughly the following:

as you can check there are some rather different reviews regarding this submission. Please feel free to comment on it or change your score in case you think it is appropriate.

There were 2 other reviews: The 1st liked the paper and gave a +2. The 2nd however had virtually the same review as mine but felt probably slightly more generous and gave a score of +1.

1st minor Question: Is it normal PC members to ask you change the review score when the review does not vary significantly from the others? I never heard such a thing, even in personal cases where was a large deviation between reviewers.

I was trying to understand the logic behind the PC message. My Theory:

I think the PC members wanted to accept the paper but because they did not have an overwhelming argument and because the average review score was lower than the acceptance threshold they were unable to justify the acceptance. Instead of altering my score at their own will, they asked me gently if I am willing to do that.

Is my theory plausible?

2nd minor Question: Since my review was virtually the same to the one of the 3rd reviewer, wouldn't it be more fair if all other parties were involved in that exchange? Maybe the 3rd reviewer would have change the score to "0" after reading my response. Why this process should involve only me?

I responded by noting the above and by standing by my initial decision and score because I really felt it did not deserve anything more than "0".

3rd Minor Question: Do you think my response was appropriate, given the circumstances? Was there any other way that I could have handled the issue?

Now, this is my final (conditional) main question:

Given the fact that my "theory" is correct and given the particular circumstances (slight variation on reviewer opinion), is it common to ask the reviewer to do their (the PC's) job of accepting/rejecting manuscripts when the reviewer's job is to provide an honest opinion? If yes, shouldn't all reviewers be aware (in case there is not some terrible misunderstanding/mistake in any of the reviews, just a matter of opinion).

Note: I was told after the event, that I was the only one that received that message. My guess is that the paper was close to acceptance but my score prevented it from being a clear accept (so they felt I was somewhat "harsh"). I guess that also explains why they did not contacted the +1 reviewer B (also to avoid situations that I change to +1 and B change to 0 after reading my justification, and we are at square 1). Any thoughts on that? Is this common?

Improve this question
6
  • 5
    It's quite common in many expert estimation tasks to ask outliers if they want to change their opinion. The PC might also have asked the +2 reviewer whether he'd like to revise his score downwards. Did you overlook a reason that made the paper valuable, or did the other reviewer overlook prior research that made the paper redundant? That's not for the PC to decide.
    – MSalters
    Commented Mar 20, 2017 at 13:46
  • @MSalters As I wrote, my review was not an outlier. It might be stricter, but that's subjective and debatable. Imagine the case where I would increase the score, and the +2 decrease it. I am fairly certain that I was the only one to be asked.
    – PsySp
    Commented Mar 20, 2017 at 13:50
  • 2
    I'm reading three scores, +0, +1, +2. That's an average of +1, but no consensus. +0 and +2 are both non-average, or outliers. As PC I'd be happy if the subject matter experts can agree on a consensus of +1. When both extremes stick to their original score, I either have to take a deeper look or just make an executive decision and use the average. (This technique is commonly used for financial and time estimates in business, where you have similar expertise challenges). So what makes you think the PC is biased?
    – MSalters
    Commented Mar 20, 2017 at 14:03
  • @MSalters I did not imply the PC is biased. I provided a theory (they wanted to accept the paper because the paper was subsequently was accepted) and under this theory it would make sense to ask me (the lower score) to increase it. Does that make sense?
    – PsySp
    Commented Mar 20, 2017 at 14:05
  • 3
    Given that there's a reasonable explanation consistent with the partial observations (ask both sides), your theory seems a bit far-fetched. For all we know, the +2 reviewer was asked the same as you, but backed up his opinion with arguments, instead of arguing against the process. Were I PC, I might then consider it a "weak +0, strong +2, take in preference over other +1's". As lawyers say: "when the facts are against you, argue the law". By arguing the process, you may have unwittingly argued against your review.
    – MSalters
    Commented Mar 20, 2017 at 14:22

1 Answer 1

Reset to default
19

First of all, I don't think there is a "conflict" as long as you don't make this one - and you really shouldn't, as there really is no indication of unethical behavior on their side.

Let's go over your questions.

Is it normal PC members to ask you change the review score when the review does not vary significantly from the others? I never heard such a thing, even in personal cases where was a large deviation between reviewers.

In many conferences, there is a PC or reviewer discussion phase wherein, after everybody has formed her or his independent impression, these impressions are compared and discussed in one way or another. Optimally, there is physical PC meeting where borderline papers are discussed, but often a mail or forum discussion will do. "Amount of deviation" is not really the right metric to decide whether a reviewer discussion will make sense, but rather whether the paper is borderline or not. There is no point in discussing whether a "Weak Accept" should be a "Strong Accept" if everybody else voted for "Strong Accept", given that the paper will likely be accepted anyway. Similarly, there is little incentive to discuss a "Weak Accept" if everybody else voted "Reject" - the paper is likely dead. For a good conference, "Borderline", "Weak Accept", and "Accept" may just put the paper on the brink where it's not entirely clear on which side it will fall.

Since my review was virtually the same to the one of the 3rd reviewer, wouldn't it be more fair if all other parties were involved in that exchange? Maybe the 3rd reviewer would have change the score to "0" after reading my response. Why this process should involve only me?

This is a good question, and in fact I think that the model of chairs communicating 1-on-1 with the reviewers is fairly ineffective (as compared to a reviewer forum, for example), although I know a few conferences that handle things like that. That being said, it makes sense to contact the person with the "lowest" rating first. Also, you don't know whether they have also contacted the other reviewers to take your review into account.

I responded by noting the above and by standing by my initial decision and score because I really felt it did not deserve anything more than "0". Do you think my response was appropriate, given the circumstances? Was there any other way that I could have handled the issue?

Sure, your answer was completely ok. Note that, as said initially, you don't have a "conflict" now with the chairs - they asked whether you think your review was too negative given the additional information of the other reviews, and you said no. That's a completely valid discussion to have, and no side acted in bad faith here. From my point of view, there is no "issue to handle", just a conversation that you will have many times over as part of the peer review process.

The only thing that might be inappropriate (depending on phrasing) was noting that it was unfair that they contacted you rather than the 3rd reviewer. As said, this is probably a mail that they also sent in similar form to 2 dozen other reviewers, and they may not quite understand what the problem is. It's not like being asked whether you want to change your rating is a personal insult.

I think it's responsibility of the PC to have valid arguments for acceptance. Is it ethical to put mild pressure on the reviewer to do their (the PC's) job of accepting/rejecting manuscripts? My job was to provide an honest opinion, which I did. My job is not to push a paper for acceptance because the PC run out of valid arguments.

I think you are overthinking this. I don't ready any pressure at all from the mailing you copied, and I have received similar mails many times over. Sometimes I changed my mind, sometimes I didn't. You provided an honest opinion, and they gave you some more info and asked whether your opinion still stands. I really don't see much agenda behind this.

Improve this answer
15
  • 1
    @PsySp I assume that there is some difference in field culture then, because in the conferences that I review for, there is some form of reviewer discussion after the normal review phase approximately for every second paper.
    – xLeitix
    Commented Mar 20, 2017 at 12:12
  • 1
    It wasn't clear to me how the reviewing was done. It sounds like @PsySp was not a PC member but an external reviewer? ... In my experience PC members will have discussion on-line and/or in person to reconcile reviews and reach consensus. It is definitely appropriate IMHO for the chair to say "there is some variance, please discuss". The suggestion that a reviewer might change their score is quite fair as well. Sometimes once you see other reviews you realize your own negative or positive sentiment didn't have all the facts. Don't see your neutral score as reason not to accept.
    – Fred Douglis
    Commented Mar 20, 2017 at 15:12
  • 1
    @PsySp, saying "I was reviewing a paper for a conference" can mean a program committee member or an external reviewer. In fact, in my experience a program chair doesn't deal with external reviewers, only the PC members who solicit reviews from external reviewers. ... That being said, we're in violent agreement here: you said it could be accepted, but you seem somehow surprised it was, or feel like they needed you to raise your score to do so. I think the comment about considering other reviews and maybe changing your score is boilerplate. Nothing unethical, or to worry about here.
    – Fred Douglis
    Commented Mar 20, 2017 at 17:16
  • 3
    I said we were in agreement!
    – Fred Douglis
    Commented Mar 20, 2017 at 17:43
  • 1
    @PsySp, my experience is that all reviewers, from the positive to the negative, get to reevaluate in the context of the full set of reviews. Sometimes it's the positive reviewer who feels less positive.
    – Fred Douglis
    Commented Mar 20, 2017 at 17:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .