When peer reviewers are public/open, are they less likely to be critical of a manuscript?

Question

I have been peer reviewing scholarly manuscripts for more than 10 years, and I am always concerned that the quality of my reviews might be impacted by the fact that my name is made public to authors or other people, instead of the typical scenario in which authors are blinded to the reviewers' identity. This is becoming more and more commonplace in the open access era.

For instance, some journals let authors but not readers know the name of the reviewers. One example is the British Medical Journal (BMJ). Specifically, the BMJ website clearly states:

"For research papers, The BMJ has fully open peer review. This means that every accepted research paper submitted from September 2014 onwards will have its prepublication history posted alongside it on thebmj.com. This prepublication history comprises all previous versions of the manuscript, the study protocol, the report from the manuscript committee meeting, the reviewers’ signed comments, and the authors’ responses to all the comments from reviewers and editors."

A similar approach, in which the peer reviewer's identity can be disclosed to authors on a voluntary basis, is followed by journals such as PLOS ONE ("Will authors know who is reviewing their manuscript? Reviewers’ identities are anonymous unless a reviewer indicates otherwise"). Other journals, instead let everybody, even readers, know their identity (e.g. World Journal of Meta-Analysis).

There is some, albeit limited, research on this (e.g. van Rooyen et al, BMJ 1999 and DeCoursey, Nature 2006).

I favor the approach followed by the Baishideng Publishing Group (fully open model), but I am wondering what is the opinion and insight of the ACADEMIA community.

Answer 1

I have a similar history of reviewing papers at around eight years, though in the fields of engineering and robotics. Between the official reviews for journals or conferences (which are almost all single blind and in rare cases double blind) and informal reviews for colleagues, I would say I differ only in level of tactfulness.

The stakes in the official review process, however, are higher and I would not agree to review if my name was to be released. I’ve rejected papers based on lack of contribution (to the extent where there were missing citations of seminal papers or even their own previous work to make the current paper look more significant) as well as lack of fundamental understanding of the involved system or experimental processes. These types of issues are not things that can be easily fixed and the review becomes more of a statement of ‘start over’ with the associated large investment of time, money, and/or resources.

Peer review requires an already rather generous donation of time by those who participate. In an open peer review there is additionally a very real fear of retribution, which only the well-established can easily ignore. My personal experience follows the results of the previously mentioned van Rooyen et al. – review quality in an open process stays approximately the same, but the number of willing reviewers decreases, potentially drastically. The open review process certainly has some value, but I don’t consider it to be widely practically implementable with the current reviewer arrangement.

Answer 2

One of the best summaries of this topic I know is from Danilo Freire. It mentions all relevant papers I came across trying to answer this question and my question about open peer review as seen from the perpective of authors.

Reading all these papers took me to the conclusion that signed reviews make a difference but that this is not perceived as inevitable negative.

Danilo Freire summarizes the studies as follows:

As noted by a number of articles on the topic, OPR creates incentives for referees to write insightful reports, or at least it has no adverse impact over the quality of reviews (DeCoursey 2006; Godlee 2002; Groves 2010; Pöschl 2012; Shanahan and Olsen 2014). In a study that used randomized trials to assess the effect of OPR in the British Journal of Psychiatry, Walsh et al. (2000) show that “signed reviews were of higher quality, were more courteous and took longer to complete than unsigned reviews.”

To clarify the quote: OPR means Open Peer Review and Danilo Freire uses "a narrow definition of OPR – only asking referees to sign their report".

Answer 3

Personally, I've reviewed for a number of journals that have let the authors know my name, or publish the name of the peer reviewers alongside the manuscript and it hasn't much changed my style. Perhaps it's because I assume you could figure my name out if you tried, or because I'm trying the "Do onto others..." approach to peer review, but there's really nothing I'd have to moderate if someone knew my name.

Which means the risk is the vindictiveness of the authors. What would likely happen is I would be somewhat pickier about who I was willing to review papers for.

I also rather like the journals going in the other direction and doing double-blind peer review.