36

A question I have been wondering for a while is if there exists an actual proof that a blind reviewing process (i.e. where the reviewers are anonymous, and the reviews not published) is better than an open one (i.e. where the reviewers are not anonymous and/or the reviews are published along with the accepted papers).

Basically, whenever I question the fact that having a blind reviewing process does not guarantee any quality (which, somehow, usually coincides with receiving a poor review for a paper ...), I'm told that anonymity is crucial for the reviewing process. But is there any proof of that? I don't believe there exists any perfect system, but I'm just not sure why does the blind (or even double-blind) one is considered as the best (or the "least worst").

Improve this question
4
  • 1
    "Better"? What does "better" mean?
    – JeffE
    Jul 14, 2014 at 16:57
  • @JeffE: good question, I'd say improving the quality, on the short and long term, of the scientific output.
    – user102
    Jul 14, 2014 at 18:53
  • An interesting question especially given that posting andcommenting in AcademicSE are not blind, while up and down votes are.
    – RoboKaren
    Jul 15, 2014 at 3:04
  • Related question about why such reviews are necessary: academia.stackexchange.com/questions/101400/…
    – Nav
    Mar 10, 2018 at 13:07

6 Answers 6

Reset to default
23

The 2008 study entitled Peer Review in Scholarly Journals - Perspective of the Scholarly Community: An International Study aimed "to measure the attitudes and behaviour of the academic community with regard to peer review." Some quotes from the summary:

Double-blind review was preferred. Although the normal experience of researchers in most fields was of single-blind review, when asked which was their preferred option, there was a preference for double-blind review, with 56% selecting this, followed by 25% for single-blind, 13% for open and 5% for post-publication review. Open peer review was an active discouragement for many reviewers, with 49% saying that disclosing their name to the author would make them less likely to review.

and

Double-blind review was seen as the most effective. Double-blind review had the most respondents (71%) who perceived it to be effective, followed (in declining order) by single-blind (52%), post-publication (37%) and open peer review (27%).

A 2008 article in Nature (and a correction) discusses the above study but the article is about double-blind review versus single-blind review, and not about blind review versus open review.

Improve this answer
8
  • 2
    This article seems good, but it seems to address the problem of blind review versus double blind, which is a bit different than blind versus open. (I actually can't access the link right now, nature.com is not responding, so I'll try to read it later).
    – user102
    Mar 7, 2012 at 11:39
  • 3
    You might also want to check out nature.com/nature/peerreview/index.html. According to the source mentioned in my previous comment, "During 2006, the journal Nature conducted a trial of open peer review; it was not a success – despite interest in the trial, only a small proportion of authors chose to participate, and only a few comments were received, many of which were not substantive. Feedback suggested ‘that there is a marked reluctance among researchers to offer open comments’."
    – JRN
    Mar 7, 2012 at 12:20
  • 1
    Thanks, the second link (publishresearch.net) addresses the question I'm interested in. Could you please reformulate your answer to include it in?
    – user102
    Mar 7, 2012 at 13:12
  • 1
    Additionally, I'll note that they don't prove that open review decreases the quality, but just that reviewers are more reluctant to review when it's open. First, if we had a model where reviewers were paid to review, they would be also reluctant to do it for free, but that doesn't mean that the free model is worst. Second, I'd be really interested in knowing how this attitude changes w.r.t. to being in post or pre-Internet generation of researcher.
    – user102
    Mar 7, 2012 at 13:15
  • 1
    Thanks for the editing, I accept the answer as I guess that's the most "objective" I can get :) But I maintain my comment that it's not shown that open reviewing decreases the quality itself, but only that people are reluctant (but I also acknowledge that it would be very hard to measure the difference of quality).
    – user102
    Mar 8, 2012 at 8:13
9

The funny thing is that on this issue, most people mention immediately the "honesty" side of the problem, not the "quality" side. For the latter, I think that this not blind/open reviews which is the pertinent question, but rather the public/private question. If reviews are always public, then my guess is that the quality will increase, because the general chair/editor in chief will push that way to ensure its conference/journal to have excellent reputation. Personally, I don't want/need to know who is reviewing my papers, but I want AND need quality reviews, and they are unusual those days. We all know why : too many papers, too many reviews to make, not that much time...

Improve this answer
2
  • That's a very good point, and that could be an effective solution. Plus, there is often valuable information in a review (such as potential related work, or different interpretations) that could be helpful for the reader.
    – user102
    Mar 8, 2012 at 8:33
  • Another potential set up this brings up, the reviewers could be blinded to the authors (to prevent dis-honesty, gender bias whatever) at the onset, but it need not be the other way around. Essentially single-blind but in the opposite way single-blind is traditionally done!
    – Andy W
    Mar 8, 2012 at 12:57
8

The more is revealed about the identity of authors and reviewers, the less honest the review process may be. Such openness may favour already-established scientists over newcomers.

In the ideal world a reviewer would raise the same concerns when reviewing papers from a Nobel-prize winner or from an undergraduate student. But as people are even afraid of asking possibly dumb questions in public, I would be really surprised if they could apply the same scrutiny regardless of who they are reviewing.

Even with the standard (single)-blind process, I heard that an already-famous scientist submitted papers under made-up names to receive honest reviews (just can't recall who).

An example from Herbert S. Wilf on a birthday speech for Donald E. Knuth (pointed out by Joel Reyes Noche):

In the 1980's, in the early days of the Journal of Algorithms, I was an Editor-in-Chief, and Don [Donald Knuth] submitted a paper to me, authored by himself under the pseudonym of Ursula N. Owens, ostensibly from some small college in some small nonexistent town in Kansas. The reason was that he really wanted to get a tough and substantive referee's report on the paper, and he had been finding that sometimes referees had pulled their punches because of his name at the top of a paper.

Double-blind process may be even more beneficial, but at the same time illusory (as topic, references and style may reveal the author). Moreover, the identity of the author may sometimes be beneficial (e.g., to compare if the new submission has something new).

None of it is a proof.

But instead of counting of lines of reviews, one can try to compare how softly (or harshly) are treated contributors, depending on their status (academic title, university name, fame/recognizability).

Improve this answer
4
  • 1
    I understand the intuition behind "the more is revealed ...", but at the same time, providing anonymity for the reviewers allow some people to write bad reviews (i.e. not reading the papers, making sloppy comments, etc), and I'd like to understand whether the pro arguments of anonymity overtake the cons ones.
    – user102
    Mar 7, 2012 at 11:37
  • Yes, I am aware that signing with ones own name is good for quality. However, for reviewing the bias and the quality are two different factors. And it is very likely that by tuning the level of 'blindness' one can trade the unbiasedness for the quality. So the issue may be harder than just the optimization of one parameter.
    – Piotr Migdal
    Mar 7, 2012 at 12:06
  • 2
    I'm not claiming it's an easy problem :) But I just don't like the argument "blindness is good, because otherwise there might be a bias. Why? Because.". As a scientist, I want hard proofs, not just intuition :)
    – user102
    Mar 7, 2012 at 12:09
  • 2
    According to a comment in mathoverflow.net/questions/45185/…, "Donald Knuth used the pseudonym Ursula N. Owens when submitting a paper to get more honest reviews. (As described by Wilf on page 3 of math.upenn.edu/~wilf/website/dek.pdf)"
    – JRN
    Mar 8, 2012 at 0:36
6

One ofhe most convincing argument I have heard in favour of blind reviewing is that it prevents people whose paper has been rejected from taking "revenge" on the reviewer (conciously or unconciously). Consider for example a senior academic who has a paper rejected because of the review of a junior academic. It would be quite easy for the senior academic to hinder the progress of the junior one.

This is similar to the point that Piotr raised about it being a more "honest" process.

I don't know of any specific examples of this though.

Improve this answer
6
  • 2
    Yes, and consider the case of a senior academic who kills a submission with a poor review either because he does not have time, or even worse, because the submission is coming from concurrents (double blinding is not that efficient), and who knows that nobody in the PC will complain about his poor review because he's senior. How is that better?
    – user102
    Mar 7, 2012 at 17:43
  • Just to be clear, I'm not claiming that the case I described is better or worse. I just don't know. I understand the arguments why blind review is good, I'd like to understand why it's the best.
    – user102
    Mar 7, 2012 at 17:47
  • I would argue that this problem would not be solved by open reviews. If you're senior enough not to care about the opinion of the leading experts in your field (i.e. the PC members), then you probably don't care about the opinion of other people either.
    – Lars Kotthoff
    Mar 7, 2012 at 17:48
  • That's a good point, but I think it's a bit different to be in a situation where only the PC chair knows that your review is not good, and might not even notice, and to publicly criticise a paper, especially when they are concurrent.
    – user102
    Mar 7, 2012 at 17:51
  • I see your point. On your previous comment, I'm not saying that blind reviewing is the best process and I think an icreasing number of people are of that opinion, too. I certainly would be happy with an open reviewing process.
    – Lars Kotthoff
    Mar 7, 2012 at 18:41
3

I don't have any proof either way, but for one data point, you might look at the HotNets workshop.

At several of the early workshops, the program committee published public reviews of the papers. During the reviewing process, program committee members wrote blind reviews, as is the usual process (the reviewers were anonymous). But also, for each published paper, a member of the program committee wrote a fresh review intended for public consumption summarizing the program committee's view on the paper. Many of the public reviews were quite frank, both in identifying reservations about the paper as well as aspects of the paper that the program committee enjoyed.

I don't think HotNets still does that, but you could try to research more about what the HotNets community's experience with public reviews was (public? negative?). Also I think there have been some other computer science workshops and conferences that have also written public reviews, so you could look at them as well.

Improve this answer
0

I can only share some of my experience.

Once a friend got a paper review request from a top journal. His boss knew that and asked him to reject it because the authors were doing what he and his boss were doing.

Improve this answer

You must log in to answer this question.