53

I'm new to the journal publishing world, and I can't but help wonder why the review process isn't completely blind? By blind, I mean the reviewers don't know who performed the research or (more importantly) what university the research came from. It seems that knowing the identity of the authors could influence the review such that the quality of the research no longer stands by itself. You should be able to read an article and, assuming that the experiment was conducted accurately and ethically, decide if it is a significant scientific advancement. Why are reviews not routinely double-blind?

0

8 Answers 8

53

In fields with a vibrant pre-print culture (e.g. Physics or Math), most papers are already publicly posted on the internet with the authors name attached before the paper is submitted and reviewed. In that context, double-blind reviewing isn't even a sensible option.

Even without pre-prints the same is true of talks in some fields. For many papers, many potential referees will have already seen a talk at a conference on the results in the paper. This may be especially true of math where the process of writing up a paper in full detail is especially onerous, so it's common for people to be giving talks on topics while the paper is in preparation.

13
  • 13
    @Behacad When there is no way the review will effectively be double-blind anyway (which is the case for mathematics), it would make journal editorial boards look extremely silly if they were to go through all the extra work (and put the authors through the extra work) involved in doing reviews double-blind. Jul 15, 2014 at 11:36
  • 14
    @Behacad I keep a feed of two subjects on arXiv, and I would be surprised to find even a few papers per year which were relevant to my work and which I had not already seen on that feed. Jul 16, 2014 at 19:19
  • 66
    Years ago, the Proceedings of the American Mathematical Society experimented with blind refereeing, and I was a blind referee for one paper. The main part of my report was essentially: "If the author is X, accept the paper. If not, put the author in touch with X, who presented the same results at conference Y." Aug 2, 2014 at 4:03
  • 7
    @AndreasBlass: That summarizes the issue so perfectly! Sep 18, 2014 at 14:29
  • 17
    A talk in math is not "work." You have to eventually write up the details. Mar 16, 2020 at 19:45
34

Theoretical computer science uses single-blind reviewing almost exclusively. Reviewers know the authors of the papers they review, but authors do not who reviews their papers. (As with many things, we copy this attitude from mathematics.)

I think the main reasons we don't use double-blind reviewing are that (1) we never have, (2) we have a habit of posting preprints (although not to the same extent as math and physics), and (3) there's a general consensus that it's just not necessary.

The standard argument that double-blind reviewing is unnecessary is that the decision to accept or reject a given paper is more objective than in other fields. There isn't an experiment to judge. Either the algorithm is faster or it isn't; either the theorem is true or it isn't; either the proof is actually a proof or it isn't. (I don't buy this argument, especially for page-limited conference submissions, but there it is.)

You should be able to read an article and, assuming that the experiment was conducted accurately and ethically, decide if it is a significant scientific advancement.

These are not the only criteria by which scientific research is judged.


Update: As @a3nm points out in his comment, theoretical computer science is slowly transitioning toward a "lightweight double-blind reviewing" protocol that is already common in other computer science research areas. "Lightweight double-blind" requires the authors to submit their papers without identifying information, citing their own work in the third person, but it does not prevent either posting preprints to arXiv or presenting work at seminars and workshops.

ALENEX, DiSC, ESA, FAccT, FODS, and LICS already follow this protocol, as do several conferences at the intersection of theoretical computer science and machine learning. Major conferences like SODA are at least seriously discussing the idea, but change is slow, and many (especially senior) researchers are strongly opposed to the idea.

For more information, see this report on double-blind reviewing at ALENEX 2018 and this FAQ from POPL 2018.

7
  • 6
    I suspect your reason (1) has a large impact. Things are slow to change. Mar 26, 2013 at 23:24
  • 2
    There is a third argument I have heard wrt theoretical computer science and our conference culture: when a program committee must make a decision within a relatively short time window about a paper on a technical subject, the paper should be scrutinized to a different extent depending on whether the authors are well-known experts. Mar 4, 2015 at 2:10
  • 7
    the paper should be scrutinized to a different extent depending on whether the authors are well-known experts — This view is not universal within theoretical computer science. (My own view is that this approach might be reasonable if its aim were to give new authors the benefit of the doubt. Unfortunately, this is usually the opposite of what actually happens.)
    – JeffE
    Mar 4, 2015 at 9:34
  • 10
    the paper should be scrutinized to a different extent depending on whether the authors are well-known experts: this is precisely the reason why we should in fact be doing double-blind reviewing in TCS. This is an obvious bias against newcomers, and precisely the kind of things that double-blind reviewing is supposed to prevent.
    – a3nm
    Mar 9, 2017 at 10:15
  • It might be different with conference papers, but in mathematics a paper (especially at a top journal) is not merely evaluated for correctness. In my limited experience, journals also ask for an evaluation of how original and how important the work is (usually offering 3–5 options for each criterion, or open ended). So the objectivity argument should be taken with a grain of salt.
    – Remy
    Aug 12, 2020 at 19:11
16

While this article by Kathryn McKinley is quite old now, it provides a much more nuanced view of the processes that would support a double blind review process. In brief, it's not as simple as "make everything double blind". There are stages where it's important, and there are stages in the review process where it's useful NOT to be double blind.

Roughly speaking, you want double blind review when doing the initial evaluation, because people are more likely to jump to conclusions on a first look. But later on, it's helpful to know who's doing the work because even in mathematical work, there's an element of trust that goes into evaluating a paper (especially in theoretical CS for example, where papers are way too short (and deadlines too close) to do a rigorous evaluation of proofs).

2
  • 1
    This aligns with a slogan I heard from the sociological study of decision making, that decision rules in the first 'screening' phase are often non-compensatory ("deal-breakers" or "deal-makers"), whereas in the second phase of closer inspection they are usually compensatory (i.e. different attributes are weighted against each other). I would cite relevant reading if I were familiar enough with the literature...
    – Remy
    Aug 12, 2020 at 19:26
  • 1
    The "element of trust" part is in line with this comment academia.stackexchange.com/questions/8891/… and gives me the same impression that this is exactly the kind of bias that double-blind reviewing should prevent.
    – a3nm
    Sep 16, 2023 at 22:16
13

The answer is at least partly that many research fields are small enough so that most people know what other researchers or research teams are doing. Anonymity is thus impossible because it does not take much to figure out who wrote the paper.

Reviewers are (in my field) often anonymous. I personally do not like this because it is easier as author to judge the comments when you know who delivered them. When you get anonymous reviews you cannot easily judge the tone if it is just the way someone expresses themselves or if it is rude (positive reviews are usually not difficult to deal with for some reason).

I have also seen in journals with open reviews how anonymous posts are full of abrasive (unprofessional) comments and hence not very constructive. I pesonally think anonymity brings out the worst in some people and openness forces one to focus on the constructive.

So, double-blind reviews can be done but I do not really see the benfit, at least not in my and related fields where communities count a few thousand.

5
  • 3
    Double-blind means that authors and reviewers are both anonymous to each other. Your complaints apply equally to single-blind reviewing, where only the reviewers are anonymous.
    – JeffE
    Mar 26, 2013 at 21:04
  • 1
    @JeffE The process involves two processes, an authored manuscript and a review, so even if the two are combined there are stil two parts with specific issues and I describe my opinion of how this would (not) work in my field. Mar 26, 2013 at 21:09
  • 1
    I agree. I wish that all submissions and reviews were entirely public and non-anonymous. Even if a reviewer can't figure out who you are it may be easy enough to figure out who you are not, and that could be enough to give space for bad behavior. I think if reviewers had to sign their names to their reviews in a completely public way this would greatly enhance the quality of reviewing and would also protect against those things that blind reviewing is supposed to protect against, but doesn't.
    – John
    Aug 1, 2014 at 18:15
  • It is little off topic, but indeed interesting how anonymity is often generate unnecessary rudeness and trolling on the side of the reviewer. Curiously, almost any other field of life, blogospheres, SE etc work pretty well in a non-anonymous way, where the reputation of a given person actually backs up its argument, and not against it.
    – Greg
    Oct 13, 2015 at 12:43
  • "I pesonally think anonymity brings out the worst in some people and openness forces one to focus on the constructive." On the other hand, anonymity forces one to consider only what one says, and not who said it.
    – Geremia
    Dec 19, 2016 at 19:41
8

As mentioned in the other responses, often double-blinding is a thin veil, anyway. Moreover, for some types of papers, knowing the authors of a paper can help assess the level of contribution.

For example, knowing that the authors of a reproducibility study are disjoint from the authors of an original paper often suggests that the original paper has truly been reproduced. In the case of an experimental Computer Science paper, this often means the code was re-implemented. This in itself is a valuable contribution (the reproduction of results) that cannot be assessed reliably from a double-blind statement like "we implement the algorithm from [7]," which may very well mean the authors just re-used the same code base that they used in [7].

As another example, knowing the authors of a study can expose their biases. Drawing again from experimental Computer Science, authors often select a subset of methods in literature against which to compare, but there is an incentive to choose ones own techniques (lower development overhead) over ones that are wildly different. (Here I wish I could find that plot over time of state-of-the-art performance relative to performance of the best chosen competitor, but I recall neither the article name, venue, nor author.)

4

In the age of Internet is not very difficult to get the name of the author (at least with a high probability), especially in a field with few practitioners. For instance see this: http://www.newappsblog.com/2012/12/the-journal-reviewing-process-isnt-anonymous-did-you-really-think-it-was-think-again.html.

So in practice only the reviewer remains anonymous.

1
  • 4
    Just because you may be able to de-anonymize the submission doesn't mean that double-blind reviewing is worthless. As a reviewer, I prefer the submission itself to be double-blind, so I'm sure I won't be biased by the author's reputation or that of their institutions; and I will not try to identify them, at least not before I have reached a final decision on the submission.
    – a3nm
    Mar 9, 2017 at 10:18
4

In some fields that involve a high degree of "practical development" (recognizeable prototypes etc.) rather than "just" theoretical concepts, it is often impractical to attempt and conceal the identity of submission authors.

A conference or journal may require submissions to be anonymized, which adds a certain extra burden on the authors: Anonymizing does not end with removing the authors list, certain sections such as an Acknowledgments section need to be removed, and parts of the text may need to be rewritten (when referencing earlier work by the same authors and would, in the final paper, write "We have already ..."). And as it is sometimes still obvious that the current submission is the direct successor of an earlier work, that reference needs to be replaced with something like Removed for double-blind review, thus naturally reducing the usefulness of the references.

And yet, all of those factors may easily be in vain, as, for example, CS often involves prototypical applications that get repeatedly extended. The theoretical possibility exists that another researcher may have gotten their hands on someone else's prototype, but when the screenshots in the submission bear a striking resemblance to those in a previously published work (*), the most likely explanation is that the two works were written by the same author or at least team.

So, as trying to reliably blind the submission is not feasible in many cases and just leads to extra work for the authors (including the discussions if the double-blind review is optional), various conferences/journals do not offer a double-blind review in the first place.

(*) I am not saying that the same contribution is published twice. I am referring to "unimportant" aspects such as the general layout of a window, or its toolbar icons, here.

3

It's not clear to me that in a small field, double-blind necssarily helps promote more impartial reviews. You gain something in removing bias/prejudice based on knowledge of the author(s). However, it becomes much more difficult to control for conflicts of interest: plenty of non-obvious connections exist between researchers that an editor can't be expected to know about. With single-blind reviewing, there is at least the opportunity for the reviewer to decline if they know they've influenced the author's views in some way, or if they feel they have reviewed too high a fraction of that author's output. If everything is double-blind, it seems quite likely that small review circles could form, to the ultimate detriment of the field.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .