(Why) Is post publication peer review (not) taking off?

Question

I know, this is a bit of a broad question. But looking at the developments and having experienced a reasonable amount of pain with established publishers and their either complex, slow, or restrictive processes/production stages, I'm tempted to ask it here and look what others think or know about that.

Post-publication peer review has been discussed quite a while ago here and here, it's been promoted and practised on platforms such as Publons and PubPeer.

But is this idea, launched many years ago to complement "classical" peer review, really successful? Are established processes/practises being replaced/improved?

Note for clarification: I envisage something like a combination of classical pre- and an optional post-publication peer review strongly based on open self-archival platforms such as arXiv. Nothing on the academic side would change, editors and reviewers would move over to such platforms to continue their work just like before, largely unpaid. Even cult-like reputation-building, gate-keeping, etc. is still possible even if not desirable. But in domains such as CS, physics, or math, where production is largely done by authors anyway, costly, error-prone, after-acceptance production stages would be replaced by just self-archival and EiC-driven quality labelling systems as they have been used for years at top-tier conferences to certify such things as reproducibility. I think, it's time for a change. (I wonder whether this question is more something for meta.)

Further note: After the many useful comments and answers, I'm extending or rephrasing my question to: Why isn't full-fledged self-archival with classical peer review and curation, post-publication commenting, and avoidance of old-school production procedures finally taking off?

Answer 1

Most papers that are published are uninteresting.

The median paper has less than one reader not counting the authors and the reviewers.

The current reviewing system relies on a network of responsibilities that are independent of the author. The editors have a responsibility to the community as a whole to get papers reviewed properly, and the community gives them esteem for performing this duty. The reviewers have a responsibility to the editor to actually do reviews, and the editor is usually some influential person (and in any case will probably at some point be the editor for one of the reviewer's papers).

Suppose I have written one of my usual mediocre papers that is among the 70% (my estimate) of papers that are never read. How would post-publication peer review be organized for it? If I have to find reviewers myself, then everyone is going to say 'No' or do a bad job, because no one feels any need to impress me and it's not a very interesting paper to read. Moreover, the best reviewers for my paper are people with a little higher status than me, and they would definitely say 'No' to me, but they might say 'Yes' to an editor. (Keep in mind that, as the system is set up, some top people in a field do far more reviewing than they write papers, while many people write one write-only paper every several years (or grad students who stop doing research write one paper from their dissertation) and never review.)

Answer 2

In a sense, it has taken off; you just need to change your definitions: in mathematics, most papers get peer reviewed after they have appeared on the Arxiv and are available to the public.

Results then appear in prestigious journals, so they are ultimately trusted and used to evaluate researchers and compare academic egos, only after they have been carefully been reviewed; but that seems working as intended and I wouldn't want it to be different.

Answer 3

In a sense, there is post-publication review: It just doesn't happen on websites, but in subsequent papers written by others. And the opinions of others are also recorded: A citation is like a "Like" on Facebook: The citing authors thought the cited paper offered some useful background to readers of their authors.

In other words, papers that are highly cited are, in some sense, positively post-publication reviewed. The converse is also true.

Answer 4

In a sense, post publication peer review has always been in place. Prior to publication peer review has the purpose of improving a paper before it appears (or at least prior to its formal publication). This is normally a private matter between a journal or conference and the authors.

Post publication, however, peer review has a different purpose and can no longe improve that paper. But anyone reading a paper can comment on it, make corrections to it and publish extensions or corrections. This has always been the case. Once a paper becomes visible it is open to comment by any "reviewer". This has, of course, let to some mighty intellectual battles, such as Leibniz v. Newton. If memory serves, Einstein's early papers weren't universally accepted but created some turmoil among the great names of the day. They generated a lot of comment, but also a lot of additional work attempting to refute or verify or extend them.

Answer 5

Because post-publication peer reviews are not currently recognized as useful contributions, so why do them when I could do something that would help me get a job?

Answer 6

As Tim Gowers wrote in 2013 in the context of an experimental peer review website:

[...] It is easy to come up with ideas for websites where people can review papers, complete with clever protocols for how the reviewing should take place, whether it is open, reward systems, etc. etc. It’s much less easy to persuade people to use the sites that are created as a result: what is going to persuade them to make the effort, when there’s only rather a small chance that the site will become in any sense “official”?

"Becoming official" is the problem here: the publishing system's main function is no longer scientific communication, no longer improving and evaluating papers. The system has been coopted by administrators and funders, in order to manage researchers' careers via bibliometric indicators. Its continued dominance makes it difficult for alternative modes of peer review to emerge, since any new initiative must compete with an entrenched, official system.

Answer 7

As in the other answers, the premise of the question is a bit inaccurate.

For work that turns out to be important, people look at it critically (with interest) long after it has supposedly been "refereed" and "published" (in the sense of being endorsed by a journal). And will look at it critically (with interest) as soon as it's available (e.g., online), whether or not it's submitted to a journal.

And, as has been true for 20+ years, there is the internet. Public availability of documents is no longer monopolized by "publishers" in the traditional sense. Of course, some people are less shy than others about putting online dubious stuff.

If one wants to be confident of the correctness of a journal document, one should probably check it in detail oneself, unless it is a very high profile case. In recent years, the guidance I've gotten as referee is that it is not my duty to check for correctness! And, well, in cases where no one cares much whether the conclusion is correct or not... ?!?

In math, in the U.S., we are still in the situation that, in most places, the only official way to score career/status/tenure/promotion points is by publication in refereed journals. People scrambling for tenure are not going to throw away their professional currency. And, in my experience, many people who've succeeded in this system are loathe to "make it easier" for the next generation. (I myself have recovered from any affection for artificial suffering... :)

Answer 8

I'm taking the definition as provided in OP:

Why isn't full-fledged self-archival with classical peer review and curation, post-publication commenting, and avoidance of old-school production procedures finally taking off?

The first part has already "taken off" and is standard in my field (physics). The latter part, post-publication commenting, has not. The reason is, why should it have taken of? For something to supplant the original, it needs to be better. Post-publication commenting is not better for several reasons:

• What would be the point? Pre-publication review can conceivably lead to the paper being improved. Post-publication review cannot.
• What are you going to say? You can't provide suggestions for improvement because the paper is already published. You could say "This paper is great, I enjoyed reading it" which is nice to read but doesn't lead to anything. Meanwhile "This paper is bad, it should never have been accepted in the first place" is painful for the authors, especially if the criticism is public, and especially if the criticism is anonymous. Example I am aware of.
• What would one gain from making the post-publication comment? Think about why people do peer review in the first place, and compare that to whether people should do post-publication commenting. Many of the reasons simply do not transfer: the results are already published so you don't get to read new exciting stuff before it's published, these things aren't organized by journals so you gain nothing from that front, and you can't say you're an acknowledged expert because odds are anyone who wants to submit a comment can do so.
• Finally, it's not a reciprocal relationship. Traditional peer review is reciprocal - you peer review for others, and in return they will peer review for you. The same does not apply for post-publication commenting.

Everyone knows the traditional peer review model is flawed, but coming up with a better alternative is pretty damn hard. Post-publication commenting is an example. You might be interested in others - off the top of my head there is open peer review, where the identity of reviewers is public, and the accept-then-review model where the journal accepts every paper before putting them up for review.

Answer 9

Post publication peer review has a model that has proven to work, in both principle and in practice: Stack Exchange. The "research question" is generally posed by someone other than the author; there are multiple short "articles" answering the research question; peer review consists of comments and, in many cases, other articles under the same question; and, ultimately, a quantitative rating is given to the quality of the question and most of the articles, with the original questioner getting a binary (checkmark) say as the "editor."

Imagine if, for example, someone posted a question on a (relevant subject) exchange to the effect of: "Is the linked article good?" Then, assuming it didn't get taken down for being off-topic or overly vague, the resulting exchange and ratings would constitute PPPR by almost any definition. (Wikipedia would be another example, but it doesn't allow original research. You might say that little on here consists of original research, but that's really only true for the quantitative, empirical sciences, and even then a lot of published journal articles just aggregate known information.)

Most of the reasons for PPPR's failure to launch, as given in other answers, are refuted by Stack Exchange. This community consists of thousands of people, including "real" experts, who are willing to read and respond to what frequently are indisputably boring or obvious questions, for free, in an unofficial forum. Some even perform basic copyediting! True, the questions and answers, individually, are shorter than an article. But Wikipedia shows that people are also willing to read and contribute to much longer articles. And sure, there's little incentive for authors to submit their work to an "unofficial" site, but as the above example shows, you don't need the author to submit it themselves.

So, either Stack Exchange has uniquely captured lightning in a bottle (in which case it should establish its own PPPR forum) or PPPR sites have failed to copy a remarkably effective format. And that's the answer to your question.