52

Every so often, I stumble upon a question concerning "double blind peer review", i.e., the authors don't know the reviewers (as usual) but also the reviewers don't know the identity of the authors.

It's always struck me as a rather odd system, with no discernible benefits (because I am under the impression that it's often rather easy to determine who the authors of a paper are anyway), and the creation of dozens of problems with people always being worried of somehow "breaking" the double blindness. (It is not helped by the fact that this would be impossible to implement in my field, math, where everyone posts their preprints on arXiv and people don't hesitate to give talks about unpublished papers).

Is there any actual, scientific, serious research into evaluating the benefits of this system? Or is it all just inertia/tradition/good intentions? I'm not talking about armchair justifications for it with no data to support it.

Improve this question
13
  • 6
    Roughly analogous question about single-blind review: Open versus Blind reviewing process
    – O. R. Mapper
    Commented Mar 9, 2018 at 12:45
  • 3
    @Mehrdad I don't frequent websites for conspiracy theorists.
    – user9646
    Commented Mar 10, 2018 at 8:10
  • 3
    "a rather odd system, with no discernible benefits" I would suggest you haven't given this minimal thought if you really believe that.
    – neuronet
    Commented Mar 10, 2018 at 14:37
  • 5
    @NajibIdrissi And yet, you come here with a conspiracy theory of your own (that double blind peer reviews are not beneficial).
    – Agent_L
    Commented Mar 10, 2018 at 16:59
  • 7
    @Agent_L That's not what I'd call a conspiracy. It's not a secret that many things continue to be just out of tradition or inertia. I'm sure you can think of a few of them in academia. The fact that actual research was conducted on the subject shows that I'm not the only one wondering.
    – user9646
    Commented Mar 10, 2018 at 17:32

4 Answers 4

Reset to default
79

Yes, there is scientific research on this topic.

Using data from ACM WSDM'17, A. Tomkins et al. showed that double blind reviewers are less likely than single blind reviewers to accept papers from famous authors, top universities, and top companies.

See: http://www.pnas.org/content/pnas/114/48/12708.full.pdf

Update: As mentioned by @thomas-supports-monica in the comments, there is some evidence that the article linked above may have some issues. See https://papers.nips.cc/paper/8770-on-testing-for-biases-in-peer-review.pdf .

Improve this answer
25
  • 9
    Does the conclusion that you cite, in itself, show that the system is beneficial? For this does one not also require an assumption that single blind reviewers accept such papers more than they "ought to"?
    – Rupe
    Commented Mar 9, 2018 at 15:38
  • 4
    @Rupe Insightful question (+1 for logic), but it seems hard to imagine this being the case.
    – jjc385
    Commented Mar 9, 2018 at 15:55
  • 27
    @Rupe To expand on other comments, it is assumed that research should be published exclusively based on the quality of its content, hence no effect should be due to the author's names and identities, and the research shows not to be the case in single blind reviews.
    – Marco Capitani
    Commented Mar 9, 2018 at 16:30
  • 4
    My initial reading of Matteo's answer was that the initial "Yes" was a response to the title. If I'd read it as "yes there is research into the question" then my logical point might not have occurred to me (and I can see that this is a better reading, since there's a comma not a full stop after "yes"). But, sticking with my logical pedantry, I'm not convinced by @Marco's point. Surely this research (as described here, at least) just shows a difference between single- and double-blind? The "effect due to author's names etc" could be there in the double-blind case and not in the single-blind.
    – Rupe
    Commented Mar 9, 2018 at 16:37
  • 11
    @Rupe: You're saying that reviewers might be biased by author names only in the case where the reviewer isn't told the author names?
    – user2357112
    Commented Mar 9, 2018 at 21:38
29

It's always struck me as a rather odd system, with no discernible benefits

Given the ample evidence that non-blinded peer review is biased against a number of different "types" of authors, including women, those with names associated with certain regions, etc. I'd argue that there's a great many discernible benefits. A few examples:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5360442/ http://science.sciencemag.org/content/sci/333/6045/925.full.pdf

etc. I'd suggest given the widespread evidence for implicit bias in a number of different fields, that the onus is on non-double-blind systems to demonstrate that they're not harmful, rather than vice versa. However, it's also been shown that single-blind reviewers preferentially favor top universities, authors and companies over double-blinded colleagues in this study: http://www.pnas.org/content/114/48/12708

Improve this answer
6
  • 4
    The question is how much double-blind-review really helps. Within my field I can identify the top researchers just by their research topic, used methods, language or how they draw figures. Sometimes even the title is enough to know who did it. And references usually tell you a lot too.
    – user64845
    Commented Mar 9, 2018 at 23:19
  • 16
    @DSVA I will note that, given I often review for a double-blind journal, I've conducted an informal evaluation of how often I correctly guess an author when I'm like "Oh, I totally know who this is...". It's below 50%. Your mileage may vary, of course, but I suspect that while people underestimate their implicit bias, they also overestimate their ability to break blinding by content alone.
    – Fomite
    Commented Mar 9, 2018 at 23:21
  • 3
    @DSVA And how does your guessing work for new authors in your research area, are you just assuming a new name couldn't possibly match the top researchers? A new name that intentionally looks to emulate the best in the field in terms of format, style, and the hot topic because they're trying to break into it.
    – ttbek
    Commented Mar 10, 2018 at 1:10
  • 1
    While the first part of your answer is interesting (even though I knew it already), it's not an answer, it's a comment. The second part of your answer is unrelated to the first part and already in the other answer (literally the same link).
    – user9646
    Commented Mar 10, 2018 at 8:11
  • @Fomite Didn't your informal evaluation entice you to try and guess who the authors were in cases where you wouldn't have tried if you hadn't been evaluating?
    – user9646
    Commented Mar 10, 2018 at 13:59
3

This isn't a scientific article, but offers a different take than the other two answers:

http://blogs.plos.org/absolutely-maybe/2017/10/31/the-fractured-logic-of-blinded-peer-review-in-journals/

Improve this answer
3

This recent study in finance/economics is relevant to the question and I found it rather striking despite some limitations. It was a field experiment that essentially had 3 conditions:

  1. Anonymous peer review
  2. Non-anonymous peer review, author is Nobel Laureate
  3. Non-anonymous peer review, author is relatively unknown post-doc

The authors listed are real people. Peer reviewers were solicited and given one of those 3 pieces of information about authorship, but in all cases the content of the manuscript was identical.

Fig. 1 in the paper tells the story. For post-doc, 66% of reviewers recommended reject, 25% major revision, 8% minor revision, 2% accept. For Nobel Laureate, 23% recommend reject, 19% major revision, 38% minor revision, 21% accept. In the anonymous case, 48% recommend rejection, 28% major revision, 22% minor revision, 2% accept. Anonymous author is treated modestly more kindly than the non-anonymous postdoc.

As a secondary finding, researchers also varied when the disclosure of author identity happened — either before or after peer reviewers accepted the invitation. Reviewers were a good bit more likely to accept the invitation when they knew they'd be reviewing the work of a Nobel Laureate.

Most obvious weakness of this study in my view is that the authors differ in more ways than the career accomplishments the study's authors emphasize; they also differ in age and race/ethnicity at minimum. For the topic of discussion here, though, it is relatively immaterial why peer reviewers are biased by knowledge of author, just that they are for whatever reason.

Note: pre-print, non-paywalled version available here.

Improve this answer

You must log in to answer this question.