What are the advantages and disadvantages of making the review content public?

Question

In websites such as publons, reviewers can declare that they reviewed papers for some journals. In addition, they are encouraged to upload the full text of their review.

On one hand, the review content might be useful for readers of the paper: it contains ideas and perspectivese that are often not visible in the final manuscript.

On the other hand, if the review content is made public, the author will know exactly who the reviewer is, which apparently harms the blind-review process.

Are there any other advantages/disadvantages to publishing review contents publicly?

Answer 1

I believe that there are many sides to this, but overall I think that the "sanctity" of blind review should be maintained. In either case, I think that at most, one can allow access to the review content, but not reveal the identities of the reviewers, nor the review scores.

While it is true that one can sometimes guess a reviewer's identity from the content of the reviews, I think that this is not an issue because

1. It's still a guess, and more importantly
2. If your review potentially reveals your identity (say by you unnecessarily suggesting all of your papers as ones that should be cited), then it's a bad review, and the practice should be discouraged.

Review scores are often useful to weed out clear accepts and clear rejects, but become really hazy when talking about borderline cases (see further comments below).

Having gotten that out of the way -

Pros of publishing reviews:

• It encourages more careful reviews: if I know that my reviews will be public I'm encouraged to make sure that they're more careful.
• It discourages abusive and unprofessional language in reviews. Ad hominem attacks in reviews do happen, and are a bad practice.
• It encourages editors/program chairs to select reviewers who do a good job. I've seen reviewers submitting a single paragraph; this is not ok.

Cons of publishing reviews:

• It encourages reviewers hold back and not write scathing/overly gushing reviews even if they truly feel strongly about the paper.

• It may render papers less readable, as authors would rely on the reviews to clarify what they should have (as Buffy mentions)

• Reviewers will worry about their identity being compromised, and thus early career researchers may not want to review at all/write very positive reviews.

• It may cause community dissent: interdisciplinary communities host several small sub-communities, each with its own publication norms and acceptance thresholds. This is already a major issue, that will be further exacerbated by revealing reviews (this may be a pro, not entirely decided!).

I think that one of the main reasons I'm against revealing reviews is because the review process, unfortunately, involves a lot of random luck. As the NIPS experiment shows, PCs can widely disagree on a lot of papers. It is important to note that there is general agreement on what are very bad papers and very good papers, but a significant chunk of them are borderline cases that could swing either way. I think that showing the reviews for borderline cases might be a bad idea, as it would cause a lot of dissent in the community ("why did my paper, with one weak reject, one weak accept and one accept, got rejected but my paper with better scores got accepted?"). I think that the "clear accept" papers can be easily listed with their reviews; the others? Not so sure...

Overall, I think that presenting reviews might be good in some cases and should involve the informed consent of both the authors and all reviewers. However, I worry about it becoming a norm. Paraphrasing a bit, review decisions are like sausages: it's better not to see how they're made.

Answer 2

the author will know exactly who the reviewer is, which apparently harms the blind-review process.

To consider the advantages and disadvantages, I think it helps to understand the motivations behind the blinding. In the case of the author not knowing the reviewer's identity, the reason is to allow the reviewer to be frank. For example, if you're an Early-Career Researcher, you might not want a more senior researcher to know you criticised their work. By hiding your identity, some of the pressure to withhold criticism is removed.

Thus, to consider whether you'd want to make your review contents public, you should consider whether you would mind the author knowing your identity, and whether that would affect how well you can do your job as a reviewer.

Answer 3

I don't have a good solution/answer, but think the following should be considered.

If you ask different publishers whether publishing blind reviews is appropriate you will likely get different answers.

The timing of publishing such reviews surely matters. If you publish reviews after final resolution of the paper's fate, compromising the blind nature is probably moot. Neither the author(s) nor the reviewer can influence the other improperly.

Longer term, if the blind review process is undermined by (too early) publishing of reviews, something will be lost and games will be played that undermine the whole process. There are valid reasons for blind reviewing. Among other things, saying something bad about a paper's methodology or results will often, I think, be misinterpreted as saying something bad about the author(s). This could have very bad long-term consequences when people start to argue in public about things that should have been settled.

My personal view is that a paper should stand on its own, without the commentary of others. If it doesn't, then it isn't a very good paper, and those issues should have been settled during review and rewrite phases.

I wouldn't publish any review, but that is just a personal choice that I'm not yet willing to recommend to others. But if it is done, I'd suggest that it be with the knowledge and permission of all parties: reviewer, author(s), publisher.

On the other hand, I don't think there is any issue about someone who previously reviewed a paper commenting in public on the final version of the paper. You would certainly be permitted to do that had you not been a reviewer.

---

For what it is worth, there is an alternate process for improving papers that is used in the patterns community to improve patterns papers. It is called the Shepherding Process (warning: PDF). The paper at the link explains the process, expressed as a pattern language. The key element is that it is not blind reviewing, but a cooperative, iterative, process to improve the paper.

Answer 4

As argued in Vincent's answer, anonymity of the reviewer is a good thing, at least in some cases.

It could be argued that making some reviews public already reveals information that harms the anonymity of all other peer reviewers: I know that prof. X would probably make his review public if he reviewed my paper, he did not, hence he is not the reviewer, and it's more likely that it's dr. Y instead. Bayes theorem and all the like. So, if we follow this argument, making any review public is harmful for the blind review process and should be avoided.