Every so often, I stumble upon a question concerning "double blind peer review", i.e., the authors don't know the reviewers (as usual) but also the reviewers don't know the identity of the authors.

It's always struck me as a rather odd system, with no discernible benefits (because I am under the impression that it's often rather easy to determine who the authors of a paper are anyway), and the creation of dozens of problems with people always being worried of somehow "breaking" the double blindness. (It is not helped by the fact that this would be impossible to implement in my field, math, where everyone posts their preprints on arXiv and people don't hesitate to give talks about unpublished papers).

Is there any actual, scientific, serious research into evaluating the benefits of this system? Or is it all just inertia/tradition/good intentions? I'm not talking about armchair justifications for it with no data to support it.

  • 6
    Roughly analogous question about single-blind review: Open versus Blind reviewing process Mar 9 '18 at 12:45
  • 3
    @Mehrdad I don't frequent websites for conspiracy theorists.
    – user9646
    Mar 10 '18 at 8:10
  • 2
    "a rather odd system, with no discernible benefits" I would suggest you haven't given this minimal thought if you really believe that.
    – neuronet
    Mar 10 '18 at 14:37
  • 5
    @NajibIdrissi And yet, you come here with a conspiracy theory of your own (that double blind peer reviews are not beneficial).
    – Agent_L
    Mar 10 '18 at 16:59
  • 5
    @Agent_L That's not what I'd call a conspiracy. It's not a secret that many things continue to be just out of tradition or inertia. I'm sure you can think of a few of them in academia. The fact that actual research was conducted on the subject shows that I'm not the only one wondering.
    – user9646
    Mar 10 '18 at 17:32

Yes, there is scientific research on this topic.

Using data from ACM WSDM'17, A. Tomkins et al. showed that double blind reviewers are less likely than single blind reviewers to accept papers from famous authors, top universities, and top companies.

See: http://www.pnas.org/content/pnas/114/48/12708.full.pdf

Update: As mentioned by @thomas-supports-monica in the comments, there is some evidence that the article linked above may have some issues. See https://papers.nips.cc/paper/8770-on-testing-for-biases-in-peer-review.pdf .

  • 8
    Does the conclusion that you cite, in itself, show that the system is beneficial? For this does one not also require an assumption that single blind reviewers accept such papers more than they "ought to"?
    – Rupe
    Mar 9 '18 at 15:38
  • 4
    @Rupe Insightful question (+1 for logic), but it seems hard to imagine this being the case.
    – jjc385
    Mar 9 '18 at 15:55
  • 25
    @Rupe To expand on other comments, it is assumed that research should be published exclusively based on the quality of its content, hence no effect should be due to the author's names and identities, and the research shows not to be the case in single blind reviews. Mar 9 '18 at 16:30
  • 4
    My initial reading of Matteo's answer was that the initial "Yes" was a response to the title. If I'd read it as "yes there is research into the question" then my logical point might not have occurred to me (and I can see that this is a better reading, since there's a comma not a full stop after "yes"). But, sticking with my logical pedantry, I'm not convinced by @Marco's point. Surely this research (as described here, at least) just shows a difference between single- and double-blind? The "effect due to author's names etc" could be there in the double-blind case and not in the single-blind.
    – Rupe
    Mar 9 '18 at 16:37
  • 10
    @Rupe: You're saying that reviewers might be biased by author names only in the case where the reviewer isn't told the author names? Mar 9 '18 at 21:38

It's always struck me as a rather odd system, with no discernible benefits

Given the ample evidence that non-blinded peer review is biased against a number of different "types" of authors, including women, those with names associated with certain regions, etc. I'd argue that there's a great many discernible benefits. A few examples:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5360442/ http://science.sciencemag.org/content/sci/333/6045/925.full.pdf

etc. I'd suggest given the widespread evidence for implicit bias in a number of different fields, that the onus is on non-double-blind systems to demonstrate that they're not harmful, rather than vice versa. However, it's also been shown that single-blind reviewers preferentially favor top universities, authors and companies over double-blinded colleagues in this study: http://www.pnas.org/content/114/48/12708

  • 3
    The question is how much double-blind-review really helps. Within my field I can identify the top researchers just by their research topic, used methods, language or how they draw figures. Sometimes even the title is enough to know who did it. And references usually tell you a lot too.
    – user64845
    Mar 9 '18 at 23:19
  • 15
    @DSVA I will note that, given I often review for a double-blind journal, I've conducted an informal evaluation of how often I correctly guess an author when I'm like "Oh, I totally know who this is...". It's below 50%. Your mileage may vary, of course, but I suspect that while people underestimate their implicit bias, they also overestimate their ability to break blinding by content alone.
    – Fomite
    Mar 9 '18 at 23:21
  • 2
    @DSVA And how does your guessing work for new authors in your research area, are you just assuming a new name couldn't possibly match the top researchers? A new name that intentionally looks to emulate the best in the field in terms of format, style, and the hot topic because they're trying to break into it.
    – ttbek
    Mar 10 '18 at 1:10
  • While the first part of your answer is interesting (even though I knew it already), it's not an answer, it's a comment. The second part of your answer is unrelated to the first part and already in the other answer (literally the same link).
    – user9646
    Mar 10 '18 at 8:11
  • @Fomite Didn't your informal evaluation entice you to try and guess who the authors were in cases where you wouldn't have tried if you hadn't been evaluating?
    – user9646
    Mar 10 '18 at 13:59

This isn't a scientific article, but offers a different take than the other two answers:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy