The following paper presents findings from a recent investigation at three major Software Engineering and Programming Languages conferences (namely, ASE, OOPSLA and PLDI 2016).
Claire Le Goues, Yuriy Brun, Sven Apel, Emery Berger, Sarfraz Khurshid, Yannis Smaragdakis: Effectiveness of Anonymization in Double-Blind Review. CoRR abs/1709.01609 (2017)
During the review process, the reviewers were urged to provide a guess if they thought they knew an author of the given paper.
On the percentage of papers where a guess was made:
For the three conferences, 70%–86% of reviews were submitted
without guesses, suggesting that reviewers typically did not believe
they knew or were not concerned with who wrote most of the papers
On the correctness of guesses:
When reviewers did guess, they were more likely to be correct
(ASE 72% of guesses were correct, OOPSLA 85%, and PLDI 74%).
However, 75% of ASE, 50% of OOPSLA, and 44% of PLDI papers
had no reviewers correctly guess even one author, and most reviews
contained no correct guess (ASE 90%, OOPSLA 74%, PLDI 81%).
On the effect of reviewer expertise on guessing:
We conclude that reviewers who considered themselves experts were more likely to
guess author identities, but were no more likely to guess correct.
On the effect of (correct and incorrect) guesses on paper acceptance:
We observed different behavior at the three conferences: ASE submissions were accepted at statistically the same rate regardless of reviewer guessing behavior. [...] OOPSLA and PLDI submissions with no guesses were less likely to be accepted (p <= 0.05) than those with at least one correct guess. PLDI submissions with no guesses were also less likely to be accepted (p <= 0.05) than submissions with all incorrect guesses.
We find that 74%–90% of reviews contain no correct guess and that reviewers
who self-identify as experts on a paper’s topic are more likely to
attempt to guess, but no more likely to guess correctly.