Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It only takes a minute to sign up.
Sign up to join this community
Is there any research/study/survey/dataset that looked at how often the reviewers correctly guess the identity of the authors when the review is double-blind?
I am aware that the answer is likely field-dependent, or even publication venue dependent. I am mostly interested in computer science, machine learning and natural language processing, but curious about other fields as well.
Conferences in programming languages are moving toward double-blind reviewing, and since the idea is debated, people are collecting some evidence, including actual peer-reviewed papers.
In particular, even when reviewers believe they guessed the authors, it turns out they are sometimes/often incorrect, as profmartinez’s answer suggests. In the citations you’ll find actual numbers; I won’t attempt a literature review myself.
This points to this literature review (done as an editorial in SIGMOD, a database venue):
This in turn cites Fisher et al. (1994), published in the Journal of the American Medical Association — hence, a peer-reviewed publication where reviewers understand statistics well:
There are also reports from program chairs evaluating statistically double-blind reviewing, by (for instance) surveying PC members for correctness of their guesses.
After many program committees, I've come to the conclusion that we grossly overestimate our ability to guess authors based on the content of a double-blind submission.
I consider it my duty as a reviewer not to get distracted by irrelevant data like authors, so I just check the paper as is. I'm sure that with a bit of guess work (and a fast search or two) I'd be able to identify most of the authors, I deliberately try no to do so.
Most areas are really closely knit, you'd be able to identify a colleague by manierisms in writing, approach elected, results used (and papers cited). And give a good shot at identifying students of colleagues, at least up to advisor.
Probably every time. Anyone that admitted to breaking the blinding though would probably face some serious negative repercussions, and so I wouldn't expect anyone to speak up or respond to a survey that would allow a good study to be done. I've never deliberately broken the blinding on a double-blind review, but I've certainly received papers that were improperly blinded where the true authors were obvious just from looking at the title page, so I had to send them back to the editor/program chair as unreviewable.
I know of no studies about this.
The following paper presents findings from a recent investigation at three major Software Engineering and Programming Languages conferences (namely, ASE, OOPSLA and PLDI 2016).
Claire Le Goues, Yuriy Brun, Sven Apel, Emery Berger, Sarfraz Khurshid, Yannis Smaragdakis: Effectiveness of Anonymization in Double-Blind Review. CoRR abs/1709.01609 (2017)
During the review process, the reviewers were urged to provide a guess if they thought they knew an author of the given paper.
On the percentage of papers where a guess was made:
For the three conferences, 70%–86% of reviews were submitted without guesses, suggesting that reviewers typically did not believe they knew or were not concerned with who wrote most of the papers they reviewed.
On the correctness of guesses:
When reviewers did guess, they were more likely to be correct (ASE 72% of guesses were correct, OOPSLA 85%, and PLDI 74%). However, 75% of ASE, 50% of OOPSLA, and 44% of PLDI papers had no reviewers correctly guess even one author, and most reviews contained no correct guess (ASE 90%, OOPSLA 74%, PLDI 81%).
On the effect of reviewer expertise on guessing:
We conclude that reviewers who considered themselves experts were more likely to guess author identities, but were no more likely to guess correct.
On the effect of (correct and incorrect) guesses on paper acceptance:
We observed different behavior at the three conferences: ASE submissions were accepted at statistically the same rate regardless of reviewer guessing behavior. [...] OOPSLA and PLDI submissions with no guesses were less likely to be accepted (p <= 0.05) than those with at least one correct guess. PLDI submissions with no guesses were also less likely to be accepted (p <= 0.05) than submissions with all incorrect guesses.
Summary:
We find that 74%–90% of reviews contain no correct guess and that reviewers who self-identify as experts on a paper’s topic are more likely to attempt to guess, but no more likely to guess correctly.
