Why don't all disciplines follow a double-blind review system?

Question

I'm new to the journal publishing world, and I can't but help wonder why the review process isn't completely blind? By blind, I mean the reviewers don't know who performed the research or (more importantly) what university the research came from. It seems that knowing the identity of the authors could influence the review such that the quality of the research no longer stands by itself. You should be able to read an article and, assuming that the experiment was conducted accurately and ethically, decide if it is a significant scientific advancement. Why are reviews not routinely double-blind?

Answer 1

In fields with a vibrant pre-print culture (e.g. Physics or Math), most papers are already publicly posted on the internet with the authors name attached before the paper is submitted and reviewed. In that context, double-blind reviewing isn't even a sensible option.

Even without pre-prints the same is true of talks in some fields. For many papers, many potential referees will have already seen a talk at a conference on the results in the paper. This may be especially true of math where the process of writing up a paper in full detail is especially onerous, so it's common for people to be giving talks on topics while the paper is in preparation.

Answer 2

Theoretical computer science uses single-blind reviewing almost exclusively. Reviewers know the authors of the papers they review, but authors do not who reviews their papers. (As with many things, we copy this attitude from mathematics.)

I think the main reasons we don't use double-blind reviewing are that (1) we never have, (2) we have a habit of posting preprints (although not to the same extent as math and physics), and (3) there's a general consensus that it's just not necessary.

The standard argument that double-blind reviewing is unnecessary is that the decision to accept or reject a given paper is more objective than in other fields. There isn't an experiment to judge. Either the algorithm is faster or it isn't; either the theorem is true or it isn't; either the proof is actually a proof or it isn't. (I don't buy this argument, especially for page-limited conference submissions, but there it is.)

You should be able to read an article and, assuming that the experiment was conducted accurately and ethically, decide if it is a significant scientific advancement.

These are not the only criteria by which scientific research is judged.

---

Update: As @a3nm points out in his comment, theoretical computer science is slowly transitioning toward a "lightweight double-blind reviewing" protocol that is already common in other computer science research areas. "Lightweight double-blind" requires the authors to submit their papers without identifying information, citing their own work in the third person, but it does not prevent either posting preprints to arXiv or presenting work at seminars and workshops.

ALENEX, DiSC, ESA, FAccT, FODS, and LICS already follow this protocol, as do several conferences at the intersection of theoretical computer science and machine learning. Major conferences like SODA are at least seriously discussing the idea, but change is slow, and many (especially senior) researchers are strongly opposed to the idea.

For more information, see this report on double-blind reviewing at ALENEX 2018 and this FAQ from POPL 2018.

Answer 3

While this article by Kathryn McKinley is quite old now, it provides a much more nuanced view of the processes that would support a double blind review process. In brief, it's not as simple as "make everything double blind". There are stages where it's important, and there are stages in the review process where it's useful NOT to be double blind.

Roughly speaking, you want double blind review when doing the initial evaluation, because people are more likely to jump to conclusions on a first look. But later on, it's helpful to know who's doing the work because even in mathematical work, there's an element of trust that goes into evaluating a paper (especially in theoretical CS for example, where papers are way too short (and deadlines too close) to do a rigorous evaluation of proofs).

Answer 4

The answer is at least partly that many research fields are small enough so that most people know what other researchers or research teams are doing. Anonymity is thus impossible because it does not take much to figure out who wrote the paper.

Reviewers are (in my field) often anonymous. I personally do not like this because it is easier as author to judge the comments when you know who delivered them. When you get anonymous reviews you cannot easily judge the tone if it is just the way someone expresses themselves or if it is rude (positive reviews are usually not difficult to deal with for some reason).

I have also seen in journals with open reviews how anonymous posts are full of abrasive (unprofessional) comments and hence not very constructive. I pesonally think anonymity brings out the worst in some people and openness forces one to focus on the constructive.

So, double-blind reviews can be done but I do not really see the benfit, at least not in my and related fields where communities count a few thousand.

Answer 5

As mentioned in the other responses, often double-blinding is a thin veil, anyway. Moreover, for some types of papers, knowing the authors of a paper can help assess the level of contribution.

For example, knowing that the authors of a reproducibility study are disjoint from the authors of an original paper often suggests that the original paper has truly been reproduced. In the case of an experimental Computer Science paper, this often means the code was re-implemented. This in itself is a valuable contribution (the reproduction of results) that cannot be assessed reliably from a double-blind statement like "we implement the algorithm from [7]," which may very well mean the authors just re-used the same code base that they used in [7].

As another example, knowing the authors of a study can expose their biases. Drawing again from experimental Computer Science, authors often select a subset of methods in literature against which to compare, but there is an incentive to choose ones own techniques (lower development overhead) over ones that are wildly different. (Here I wish I could find that plot over time of state-of-the-art performance relative to performance of the best chosen competitor, but I recall neither the article name, venue, nor author.)

Answer 6

In the age of Internet is not very difficult to get the name of the author (at least with a high probability), especially in a field with few practitioners. For instance see this: http://www.newappsblog.com/2012/12/the-journal-reviewing-process-isnt-anonymous-did-you-really-think-it-was-think-again.html.

So in practice only the reviewer remains anonymous.

Answer 7

In some fields that involve a high degree of "practical development" (recognizeable prototypes etc.) rather than "just" theoretical concepts, it is often impractical to attempt and conceal the identity of submission authors.

A conference or journal may require submissions to be anonymized, which adds a certain extra burden on the authors: Anonymizing does not end with removing the authors list, certain sections such as an Acknowledgments section need to be removed, and parts of the text may need to be rewritten (when referencing earlier work by the same authors and would, in the final paper, write "We have already ..."). And as it is sometimes still obvious that the current submission is the direct successor of an earlier work, that reference needs to be replaced with something like Removed for double-blind review, thus naturally reducing the usefulness of the references.

And yet, all of those factors may easily be in vain, as, for example, CS often involves prototypical applications that get repeatedly extended. The theoretical possibility exists that another researcher may have gotten their hands on someone else's prototype, but when the screenshots in the submission bear a striking resemblance to those in a previously published work (*), the most likely explanation is that the two works were written by the same author or at least team.

So, as trying to reliably blind the submission is not feasible in many cases and just leads to extra work for the authors (including the discussions if the double-blind review is optional), various conferences/journals do not offer a double-blind review in the first place.

(*) I am not saying that the same contribution is published twice. I am referring to "unimportant" aspects such as the general layout of a window, or its toolbar icons, here.

Answer 8

It's not clear to me that in a small field, double-blind necssarily helps promote more impartial reviews. You gain something in removing bias/prejudice based on knowledge of the author(s). However, it becomes much more difficult to control for conflicts of interest: plenty of non-obvious connections exist between researchers that an editor can't be expected to know about. With single-blind reviewing, there is at least the opportunity for the reviewer to decline if they know they've influenced the author's views in some way, or if they feel they have reviewed too high a fraction of that author's output. If everything is double-blind, it seems quite likely that small review circles could form, to the ultimate detriment of the field.