50

I'm new to the journal publishing world, and I can't but help wonder why the review process isn't completely blind? By blind, I mean the reviewers don't know who performed the research or (more importantly) what university the research came from. It seems that knowing the identity of the authors could influence the review such that the quality of the research no longer stands by itself. You should be able to read an article and, assuming that the experiment was conducted accurately and ethically, decide if it is a significant scientific advancement. Why are reviews not routinely double-blind?

46

In fields with a vibrant pre-print culture (e.g. Physics or Math), most papers are already publicly posted on the internet with the authors name attached before the paper is submitted and reviewed. In that context, double-blind reviewing isn't even a sensible option.

Even without pre-prints the same is true of talks in some fields. For many papers, many potential referees will have already seen a talk at a conference on the results in the paper. This may be especially true of math where the process of writing up a paper in full detail is especially onerous, so it's common for people to be giving talks on topics while the paper is in preparation.

  • 3
    I don't think the argument that "review often can't be double-blind" is an argument against the benefits of using double-blind review. Just because some articles in some disciplines sometimes would already be published with a name, does not mean that the journals cannot try to do double-blind review. – Behacad Jul 14 '14 at 17:31
  • 8
    @Behacad When there is no way the review will effectively be double-blind anyway (which is the case for mathematics), it would make journal editorial boards look extremely silly if they were to go through all the extra work (and put the authors through the extra work) involved in doing reviews double-blind. – Tobias Kildetoft Jul 15 '14 at 11:36
  • 10
    @Behacad I keep a feed of two subjects on arXiv, and I would be surprised to find even a few papers per year which were relevant to my work and which I had not already seen on that feed. – Tobias Kildetoft Jul 16 '14 at 19:19
  • 43
    Years ago, the Proceedings of the American Mathematical Society experimented with blind refereeing, and I was a blind referee for one paper. The main part of my report was essentially: "If the author is X, accept the paper. If not, put the author in touch with X, who presented the same results at conference Y." – Andreas Blass Aug 2 '14 at 4:03
  • 4
    @AndreasBlass: That summarizes the issue so perfectly! – Nate Eldredge Sep 18 '14 at 14:29
30

Theoretical computer science uses single-blind reviewing almost exclusively. Reviewers know the authors of the papers they review, but authors do not who reviews their papers. (As with many things, we copy this attitude from mathematics.)

I think the main reasons we don't use double-blind reviewing are that (1) we never have, (2) we have a habit of posting preprints (although not to the same extent as math and physics), and (3) there's a general consensus that it's just not necessary.

The standard argument that double-blind reviewing is unnecessary is that the decision to accept or reject a given paper is more objective than in other fields. There isn't an experiment to judge. Either the algorithm is faster or it isn't; either the theorem is true or it isn't; either the proof is actually a proof or it isn't. (I don't entirely buy this argument, especially for page-limited conference submissions, but there it is.)

You should be able to read an article and, assuming that the experiment was conducted accurately and ethically, decide if it is a significant scientific advancement.

These are not the only criteria by which scientific research is judged.

  • 6
    I suspect your reason (1) has a large impact. Things are slow to change. – Federico Poloni Mar 26 '13 at 23:24
  • 2
    There is a third argument I have heard wrt theoretical computer science and our conference culture: when a program committee must make a decision within a relatively short time window about a paper on a technical subject, the paper should be scrutinized to a different extent depending on whether the authors are well-known experts. – Sasho Nikolov Mar 4 '15 at 2:10
  • 6
    the paper should be scrutinized to a different extent depending on whether the authors are well-known experts — This view is not universal within theoretical computer science. (My own view is that this approach might be reasonable if its aim were to give new authors the benefit of the doubt. Unfortunately, this is usually the opposite of what actually happens.) – JeffE Mar 4 '15 at 9:34
  • 6
    the paper should be scrutinized to a different extent depending on whether the authors are well-known experts: this is precisely the reason why we should in fact be doing double-blind reviewing in TCS. This is an obvious bias against newcomers, and precisely the kind of things that double-blind reviewing is supposed to prevent. – a3nm Mar 9 '17 at 10:15
12

The answer is at least partly that many research fields are small enough so that most people know what other researchers or research teams are doing. Anonymity is thus impossible because it does not take much to figure out who wrote the paper.

Reviewers are (in my field) often anonymous. I personally do not like this because it is easier as author to judge the comments when you know who delivered them. When you get anonymous reviews you cannot easily judge the tone if it is just the way someone expresses themselves or if it is rude (positive reviews are usually not difficult to deal with for some reason).

I have also seen in journals with open reviews how anonymous posts are full of abrasive (unprofessional) comments and hence not very constructive. I pesonally think anonymity brings out the worst in some people and openness forces one to focus on the constructive.

So, double-blind reviews can be done but I do not really see the benfit, at least not in my and related fields where communities count a few thousand.

  • 2
    Double-blind means that authors and reviewers are both anonymous to each other. Your complaints apply equally to single-blind reviewing, where only the reviewers are anonymous. – JeffE Mar 26 '13 at 21:04
  • 1
    @JeffE The process involves two processes, an authored manuscript and a review, so even if the two are combined there are stil two parts with specific issues and I describe my opinion of how this would (not) work in my field. – Peter Jansson Mar 26 '13 at 21:09
  • 1
    I agree. I wish that all submissions and reviews were entirely public and non-anonymous. Even if a reviewer can't figure out who you are it may be easy enough to figure out who you are not, and that could be enough to give space for bad behavior. I think if reviewers had to sign their names to their reviews in a completely public way this would greatly enhance the quality of reviewing and would also protect against those things that blind reviewing is supposed to protect against, but doesn't. – John Aug 1 '14 at 18:15
  • It is little off topic, but indeed interesting how anonymity is often generate unnecessary rudeness and trolling on the side of the reviewer. Curiously, almost any other field of life, blogospheres, SE etc work pretty well in a non-anonymous way, where the reputation of a given person actually backs up its argument, and not against it. – Greg Oct 13 '15 at 12:43
  • "I pesonally think anonymity brings out the worst in some people and openness forces one to focus on the constructive." On the other hand, anonymity forces one to consider only what one says, and not who said it. – Geremia Dec 19 '16 at 19:41
12

While this article by Kathryn McKinley is quite old now, it provides a much more nuanced view of the processes that would support a double blind review process. In brief, it's not as simple as "make everything double blind". There are stages where it's important, and there are stages in the review process where it's useful NOT to be double blind.

Roughly speaking, you want double blind review when doing the initial evaluation, because people are more likely to jump to conclusions on a first look. But later on, it's helpful to know who's doing the work because even in mathematical work, there's an element of trust that goes into evaluating a paper (especially in theoretical CS for example, where papers are way too short (and deadlines too close) to do a rigorous evaluation of proofs).

7

As mentioned in the other responses, often double-blinding is a thin veil, anyway. Moreover, for some types of papers, knowing the authors of a paper can help assess the level of contribution.

For example, knowing that the authors of a reproducibility study are disjoint from the authors of an original paper often suggests that the original paper has truly been reproduced. In the case of an experimental Computer Science paper, this often means the code was re-implemented. This in itself is a valuable contribution (the reproduction of results) that cannot be assessed reliably from a double-blind statement like "we implement the algorithm from [7]," which may very well mean the authors just re-used the same code base that they used in [7].

As another example, knowing the authors of a study can expose their biases. Drawing again from experimental Computer Science, authors often select a subset of methods in literature against which to compare, but there is an incentive to choose ones own techniques (lower development overhead) over ones that are wildly different. (Here I wish I could find that plot over time of state-of-the-art performance relative to performance of the best chosen competitor, but I recall neither the article name, venue, nor author.)

5

In the age of Internet is not very difficult to get the name of the author (at least with a high probability), especially in a field with few practitioners. For instance see this: http://www.newappsblog.com/2012/12/the-journal-reviewing-process-isnt-anonymous-did-you-really-think-it-was-think-again.html.

So in practice only the reviewer remains anonymous.

  • 1
    Just because you may be able to de-anonymize the submission doesn't mean that double-blind reviewing is worthless. As a reviewer, I prefer the submission itself to be double-blind, so I'm sure I won't be biased by the author's reputation or that of their institutions; and I will not try to identify them, at least not before I have reached a final decision on the submission. – a3nm Mar 9 '17 at 10:18
5

In some fields that involve a high degree of "practical development" (recognizeable prototypes etc.) rather than "just" theoretical concepts, it is often impractical to attempt and conceal the identity of submission authors.

A conference or journal may require submissions to be anonymized, which adds a certain extra burden on the authors: Anonymizing does not end with removing the authors list, certain sections such as an Acknowledgments section need to be removed, and parts of the text may need to be rewritten (when referencing earlier work by the same authors and would, in the final paper, write "We have already ..."). And as it is sometimes still obvious that the current submission is the direct successor of an earlier work, that reference needs to be replaced with something like Removed for double-blind review, thus naturally reducing the usefulness of the references.

And yet, all of those factors may easily be in vain, as, for example, CS often involves prototypical applications that get repeatedly extended. The theoretical possibility exists that another researcher may have gotten their hands on someone else's prototype, but when the screenshots in the submission bear a striking resemblance to those in a previously published work (*), the most likely explanation is that the two works were written by the same author or at least team.

So, as trying to reliably blind the submission is not feasible in many cases and just leads to extra work for the authors (including the discussions if the double-blind review is optional), various conferences/journals do not offer a double-blind review in the first place.

(*) I am not saying that the same contribution is published twice. I am referring to "unimportant" aspects such as the general layout of a window, or its toolbar icons, here.

2

It's not clear to me that in a small field, double-blind necssarily helps promote more impartial reviews. You gain something in removing bias/prejudice based on knowledge of the author(s). However, it becomes much more difficult to control for conflicts of interest: plenty of non-obvious connections exist between researchers that an editor can't be expected to know about. With single-blind reviewing, there is at least the opportunity for the reviewer to decline if they know they've influenced the author's views in some way, or if they feel they have reviewed too high a fraction of that author's output. If everything is double-blind, it seems quite likely that small review circles could form, to the ultimate detriment of the field.

protected by Alexandros Mar 12 '18 at 22:43

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.