26

I was reading the other day this blog post, about "how to peer review", and one passage struck me:

Don't review like a grad student

Reviews written by graduate students are among the most negative.

Grad students are often on the receiving end of negative reviews, because they are just learning how to write papers.

So, even though I'm no longer a grad student, I sometimes have the feeling that I tend to be a more negative reviewer than others, but it's actually hard to measure (maybe I'm just reviewing mostly bad papers?!), since I don't always have access to the other reviews of the papers I'm reviewing (due to the fact that I've been more often acting as an external reviewer than as a PC member).

I believe that the quality of my writing has improved with the feedback I've received from the reviewers of my paper, but because I don't have any feedback on my reviews, it's hard to know if and how I can improve them. So my question is: is there a way to measure my own reviewing bias?

The first thing that would come to my mind would be a set of papers reviewed by many other reviewers, and a result like: "you're in the x% more negative reviewers". Of course, I'm not implying that I would be automatically less severe when reviewing, but sometimes, in case of doubt, it's good to know one's own bias.

Improve this question
2
  • 3
    I’ll add a comment here because there are so many things I disagree with in the blog post you linked to… I don't think justified, documented criticism is necessarily bad. I value criticism as long as it is constructive, and I welcome external review even when it's rather hard to hear. (And, many times I have said to myself upon reading the review the second time: “duh, should have thought about that”.) Never be afraid to rewrite a paper, gather some more data, or reply to the editor making a stand and arguing why you do not agree with the reviewer's assessment…
    – F'x
    Nov 21, 2012 at 17:03
  • @F'x: I'm not saying I'm agreeing entirely with the post, but I'm not sure where it said that justified, documented criticism is necessarily bad. It actually advises to provide constructive criticism.
    – user102
    Nov 21, 2012 at 18:30

4 Answers 4

Reset to default
15

There aren’t many ways for your to evaluate how your reviews stand against others, but there is at least one. You can keep a tab of the papers you have reviewed, and look for them a year or so after the review. See if your reviews correlate with the publication (or lack thereof) of the papers in the respective journals… if papers you have suggested to reject are frequently accepted by the editor, then you give typically more negative reviews than other reviewers of your community.

Of course, this is a very black-and-white picture (rejection/publication). But, I don't think you can actually do much better, because “negativity” can hardly be measured quantitatively.

PS: I wouldn't call it a reviewing bias. A review is the editor asking you your frankly assessment of a publication. Toning it up or down because you fell that you do not conform to the “mean” of your field would probably diminish your value as a reviewer. If the editor does not think your reviews are helpful to him, he’ll just stop sending you paper to review!

I'll finish with an anecdote: there's a journal for which I have recently given a series of quite negative reviews (well documented, and justified in my eye), and I have actually received some positive feedback about my reviews from the editor.

Improve this answer
2
  • 1
    I try to keep track of the paper I've reviewed, but one thing is that most reviews I'm doing are for conferences, where it can be the case that papers with "negative" reviews are accepted (if the others have even worst reviews).
    – user102
    Nov 21, 2012 at 18:32
  • And I call it a bias more in the sense that the age or experience can have an impact of the overall review process. So I don't necessarily know if I'm too negative because that's just the way I am, or if because I'm biased :)
    – user102
    Nov 21, 2012 at 18:34
4

Two possibilities, both not ideal:

1) If the paper is revised and resubmitted, it often comes along with detailed replies to each reviewer as to how the authors addressed each review. You may get at least these replies to the other reviews, which will allow you to judge whether they were more or less severe than you. In addition, editors often send the reviews back to the author in ascending order of severity, so "reviewer 1" will be the most enthusiastic and "reviewer 3" the least. If you are significantly more often "reviewer 3" than "reviewer 1", this may say something.

2) You can always gather anecdotal evidence by simply asking the editor to provide a little feedback on your review. Of course, the best of all editors provide this feedback (in a constructive manner) without having to be asked.

Improve this answer
3
  • 1
    “editors often send the reviews back to the author in ascending order of severity” — I was very surprised by this assertion, so I checked in the reviews of my own papers and this is absolutely not true for journals of the American Physical Society, American Institute of Physics and American Chemical Society. Maybe some editors do it, but that's very far from a general rule…
    – F'x
    Nov 21, 2012 at 22:00
  • OK, this may differ by field. This practice certainly holds in psychology (perhaps psych editors like to soften the blow?).
    – Stephan Kolassa
    Nov 21, 2012 at 22:03
  • @StephanKolassa: as I said, I review mostly for conferences, where most of the time there is only one round of review, and you don't always access to the other reviews. I've also reviewed papers for some journals, but there wasn't any obvious ordering of the reviewers, although I didn't have access to the actual score, only the general review.
    – user102
    Nov 22, 2012 at 8:49
3

I'll tackle the other question: "maybe I'm just reviewing mostly bad papers?!"

I think with the same "right" that lets you ask for your personal bias, you may also ask whether there is an "assignment bias":
Possibly the "nice" papers are reviewed by the more famous people (who refuse to review everything that does not sound extremely interesting because they are always suggested) leaving the not-so-exciting-looking stuff to younger not so famous researchers.

However, the first thing is maybe to calculate whether you really do more negative reviews.

Assuming a jounal accepts 1/3 of the manuscripts, and has the policy that if the first 2 reviews conflict, a third is done. Majority wins. This situation is in accordance with 50 - 70% of the reviews being negative:

  • 50%: PP for the accepted, PNN for the 2 rejected thirds
  • 70%: PPN for the accepted, 2 x NN for the rejected

Do you return less than 2 positive for every 5 negative reviews (or whatever is the acceptance rate for the jounals you review for)?

Improve this answer
2
  • It's a good point, but it's actually hard to say, because I'm mostly reviewing for conferences, and it's often the case that I don't know the other review, and it's also possible that a paper is accepted with a majority of "negative" reviews.
    – user102
    Nov 24, 2012 at 12:31
  • @CharlesMorisset: hmm, that makes things even more difficult. In my field, we're more focussed on journal papers. Conference proceedings are counted basically as "without peer-review". Many conferences now don't publish proceedings, but a special issue in a journal, and the papers undergo the normal peer-review process of the journal.
    – cbeleites unhappy with SX
    Nov 24, 2012 at 17:38
2

is there a way to measure my own reviewing bias?

Why should even think about such a thing, like "reviewing bias"? Isn't scientific pursuit striving for the maxim of pushing the knowledge of humankind? The only thing we should care for then is whether our critique provided in the review is valid, whether its underlying argumentation is correct, to the point and appropriate with respect to the objectives of the field we are in, etc. - that is, as objective as we can do at the moment. There shouldn't be any bias involved. A review should strive for a pure argument why the presented work advances the state of the art (which should be as close as possible to "has an impact on a scientific pursuit in a given field"), or it doesn't.

Now having said that, of course even for the same honest review you can sometimes come up with two different final recommendations in the case the paper is borderline. In that case, I consider whether I myself could fix the paper to a clearly acceptable state. If so, I provide the constructive advice to the authors and lean rather to the positive side. If not, the critique stands. We are not in this game to play our career games. We are here to advance the knowledge of the humankind. Point. Of course, sometimes emotions tend to overrule our reason, if that is the case, the reviewer is most probably in a conflict of interest.

I would strongly oppose to "measure" ones "reviewing bias". We should rather measure correctness and validity of our argumentation for, or against the presented submission. If 100 reviewers of a given paper give it a thumbs up, but you can by no means agree or get in line with their argumentation, why should you try to accommodate? Wouldn't that defeat the purpose of peer-review? I am not saying we shouldn't learn from such encounters of course.

Disclaimer: I really appreciate your question. Of course I do share your concerns very much and very often. But it helps me to put the maxim stated above before this concern.

Improve this answer
2
  • "There shouldn't be any bias involved" ... well, the point of the bias, is that you don't really choose to have one or not :) Of course, I value the fact that different people have different opinions, but I was just interested the argument "grad students usually receive negative reviews, because they just start, so they tend to write negative reviews", and because I'm a scientist, I like to measure things rather than believe in established ideals ;)
    – user102
    Nov 22, 2012 at 8:54
  • @CharlesMorisset: a possible explanation for the leaning of grad students to the negative side may be their tunnel vision and lack of broader understanding of the topic. I myself was stung by this in the past. Fortunately, most CS/AI conferences I am reviewing for have an open discussion phase where an incorrect review gets trashed in a very straightforward manner. The bigger ones also have a author rebuttal phase, so as a reviewer I get quite a lot of feedback on my reviews. These are good for sharpening ones knives and swords over time :-).
    – walkmanyi
    Nov 22, 2012 at 10:33

You must log in to answer this question.