Risks of shifting to post-publication review

Question

I've heard some compelling arguments from open science advocates for a system of reviewing scientific research outputs that focuses almost exclusively on post-publication peer review. The idea is that research would initially be published in free and open venues, with little to no lag time resulting from pre-publication review (pre-pub review, if any, is focused on weeding out large and obvious flaws and not on assessing perceived significance or impact). Once published, social tools similar to those in place at Amazon and StackOverflow (reviewing, commenting, reputation systems, etc.) would be used by the community for more in-depth assessment and critique of research outputs. Scientists could get credit for post-publication reviews, which (if openly accessible along with the article) are valuable scientific contributions themselves. On the other hand, junior scientists afraid of unjust retribution for being critical of a senior scientist's research could post reviews anonymously/pseudonymously, protecting their professional reputation but still making an important contribution to the scientific discourse.

I've seen two primary arguments against this kind of system: 1) that pre-publication review is an important quality control mechanism, and 2) that it ensures that the published scientific literature actually gets read. Regarding this second point, many raise the point that many research articles go uncited (and potentially unread by anyone other than the authors and reviewers). It could be tragic if a major scientific breakthrough existed in the literature, but was never widely acknowledged because the paper never got any attention and post-publication review.

Do these risks give real cause for concern? Is there some level of quality control that can only be done pre-publication? Do we risk missing out on important research by not requiring rigorous review of all published material? And are there other risks not mentioned here that would make a shift from primarily pre-pub review to primarily post-pub review problematic?

Answer 1

In my mind, there are three semi-related problems with post-publication peer review:

1. It's difficult for the lay press to distinguish between pre- and post-publication reviewed papers, and for many "breaking" articles, I'd be concerned about reports of spurious results becoming more common, not less. For example, if I publish a paper on a risk factor for X disease, how long does the press have to wait to report it? Do they have to let peer reviews build up? Pre-pub at least ensures that someone has looked it over.
2. Article readership is nowhere near uniform, and there's no reason to believe reviews would be different. At the moment, every single paper published by a journal is subjected to some form of review, whether anyone else cares about that paper or not. Relying on post-publication, it's possible some papers will get few/no reviews. What's a non-expert to make of those papers? Similarly, what's a non-expert to make of a post-pub vaccine study that's been flooded with anti-vaxx rhetoric dressed up in technical jargon?
3. Post-pub peer review is inherently non-anonymous. I'm pretty sure Big Deal Figure in the Field is not going to be judged as harshly as New Grad Student even for the same work - and this will also likely effect those with clearly female or minority "sounding" names. Prominent scientists may be more likely to get a pass under the assumption that they're right. In my mind, breaking review anonymity is a step in the wrong direction.

Answer 2

The function of the editor in a regular reviewed journal is to determine whether a submission meets the intellectual standards of that journal. The standards are in fact largely established by the editor, via explicit policy statements, choice of reviewers, and the editor's power to make final decisions that are contrary to the opinion of the majority of reviewers. If one were to replace carefully-focused criticism by a small number of well-chosen experts with a count of "likes" within an undefined "community" (e.g. Stackexchange / Reddit style voting), the valuable contribution of the few experts who write detailed comments would probably be nullified by the vast number of uninformed voters who are really in no position to render a professional opinion about the paper.

It is extremely difficult to persuade qualified experts to spend the time reviewing journal submissions, and such "credit" as is attached to reviewing comes in the form of a line in the "service" entry in your annual report, where you attest to having reviewed mss. for various journals; or, even worse, your skills as a reviewer may be recognized and you will be invited to be an editor or associate editor. One fact that motivates quite a number of reviewers is not the "credit" for reviewing, but the opportunity to significantly influence the course of the field. This comes about in the review system both by giving the reviewer a privileged opportunity to nudge the research in a certain direction (e.g. pointing out that the facts are compatible with multiple theories; pointing out that the logic of the argument doesn't support A, it supports B), and by putting them in the position of absolutely preventing the publication of a bad paper (recommending reject, and doing so persuasively). When rejection is a possible outcome, authors will take criticisms seriously.

In a post-publication review system, this reviewer incentive would be missing. What, then would be the motivation for an expert to write an after-the-fact review of a terrible paper, so that the world at large would know that the paper is terrible? And without some reason to think that a paper is worth reading, why would you bother reading this one when there are hundreds of others coming out every month?

Your suggested post-publication review essentially describes a blog. Perhaps it works in some fields, but in mine, nobody puts serious research in a blog.

Answer 3

Regardless of whether the end-result would be better, one problematic part of a shift is always the shift. There are a lot of processes in place based around the current system, and it's very unlikely all stakeholders will simultaneously be in favour of change. Even if all academics agreed that moving to post-publication would be beneficial, there's a good chance that those allocating jobs, promotions, grants, league-table positions, student funding... would still only consider traditional journals. That could make the cost of change too high for those who, for example, need to support their families.

I would say that the UK open-access situation is an example of this. The government decided it liked the idea of open-access, but then realised that the cost of transition from reader-pays to author-pays was higher than it was prepared to pay right now.

As an aside, there have been historical examples (at least within maths of)

a major scientific breakthrough existed in the literature, but was never widely acknowledged

Answer 4

The dark side of post-publication peer review

The main issue raised here seems to be the problem of policing who gets to comment on an article, and how they are identified. Anonymity is important to allow good criticism to be made, but also opens the flood-gates for trouble.