I know, this is a bit of a broad question. But looking at the developments and having experienced a reasonable amount of pain with established publishers and their either complex, slow, or restrictive processes/production stages, I'm tempted to ask it here and look what others think or know about that.

Post-publication peer review has been discussed quite a while ago here and here, it's been promoted and practised on platforms such as Publons and PubPeer.

But is this idea, launched many years ago to complement "classical" peer review, really successful? Are established processes/practises being replaced/improved?

Note for clarification: I envisage something like a combination of classical pre- and an optional post-publication peer review strongly based on open self-archival platforms such as arXiv. Nothing on the academic side would change, editors and reviewers would move over to such platforms to continue their work just like before, largely unpaid. Even cult-like reputation-building, gate-keeping, etc. is still possible even if not desirable. But in domains such as CS, physics, or math, where production is largely done by authors anyway, costly, error-prone, after-acceptance production stages would be replaced by just self-archival and EiC-driven quality labelling systems as they have been used for years at top-tier conferences to certify such things as reproducibility. I think, it's time for a change. (I wonder whether this question is more something for meta.)

Further note: After the many useful comments and answers, I'm extending or rephrasing my question to: Why isn't full-fledged self-archival with classical peer review and curation, post-publication commenting, and avoidance of old-school production procedures finally taking off?

  • 2
    Editors and reviewers are unpaid, but the editor's secretary, who keeps track of who all the reviewers are and whether they need a reminder or not and tracks down the reviewer who changed jobs but forgot to tell the editor their new e-mail address (software can help with this but can't do all of it) has to be paid. Commented May 6, 2021 at 21:54
  • One needs a mechanism to encourage such reviews to be collected. For example, all referees/reviewers for awards and promotions could be requested to review at least one paper in order for their letter to carry additional weight.
    – Kapil
    Commented May 7, 2021 at 4:46
  • 1
    We need a robust reddit-style comment system on the Arxiv so that people can discuss papers easily and openly. The comments will be very enlightening. The comment system should support upvoting and downvoting comments and should support mathjax. This needs to exist.
    – littleO
    Commented May 8, 2021 at 2:56
  • @littleO aren't conferences and coffee breaks existing to do that? plus, having a written trace or a permanent record actually prevents open and easy discussion. Academia is power unbalance, retaliation fear is too high, try to naively hit a weak point of one of the best recognized professor in your field as a PhD and you will see how long it will take you to recover if he/she is a bad-temper person...
    – EarlGrey
    Commented Dec 7, 2021 at 9:29

9 Answers 9


Most papers that are published are uninteresting.

The median paper has less than one reader not counting the authors and the reviewers.

The current reviewing system relies on a network of responsibilities that are independent of the author. The editors have a responsibility to the community as a whole to get papers reviewed properly, and the community gives them esteem for performing this duty. The reviewers have a responsibility to the editor to actually do reviews, and the editor is usually some influential person (and in any case will probably at some point be the editor for one of the reviewer's papers).

Suppose I have written one of my usual mediocre papers that is among the 70% (my estimate) of papers that are never read. How would post-publication peer review be organized for it? If I have to find reviewers myself, then everyone is going to say 'No' or do a bad job, because no one feels any need to impress me and it's not a very interesting paper to read. Moreover, the best reviewers for my paper are people with a little higher status than me, and they would definitely say 'No' to me, but they might say 'Yes' to an editor. (Keep in mind that, as the system is set up, some top people in a field do far more reviewing than they write papers, while many people write one write-only paper every several years (or grad students who stop doing research write one paper from their dissertation) and never review.)

  • 5
    Yup! Papers are awful and refereeing is the worst. There is one other motivating factor here that you don’t mention, I referee young mathematicians work (even if I don’t feel obligated to the editor) because I want them to get jobs. But that mostly applies just to postdocs. Commented May 6, 2021 at 15:57
  • 3
    Note that post-publication (or non-publication) peer review happens pretty well for interesting papers that people want to read. Perelman’s work went through a very extensive process to verify the work and expand on the details despite never being published. Conversely, everyone knows where the hole is in Mochizuki’s work from outside readers, even though it’s published and the referees didn’t do their job. Commented May 6, 2021 at 16:03
  • 15
    Actually, it isn't that most papers are uninteresting. It is that the audience for extremely specialized work is limited in many fields.
    – Buffy
    Commented May 6, 2021 at 21:53
  • 8
    Do you have a source for The median paper has less than one reader not counting the authors and the reviewers? I can ask a separate question on this if you consider it off-topic for this one.
    – Allure
    Commented May 7, 2021 at 6:10
  • 3
    @SylvainRibault: Because research is not the only purpose of research. I know someone with a PhD who got their bachelor's degree from Alma College. There, the professors teach 360 classroom hours or more a year, leaving little time for research. Alma College and the mathematical community would like to know that the professors there have a good sense of what mathematical research is so that we can trust the recommendation letters they write (and for other purposes). So we want to review their occasional papers even though they don't seriously advance mathematics. Commented May 7, 2021 at 21:14

In a sense, it has taken off; you just need to change your definitions: in mathematics, most papers get peer reviewed after they have appeared on the Arxiv and are available to the public.

Results then appear in prestigious journals, so they are ultimately trusted and used to evaluate researchers and compare academic egos, only after they have been carefully been reviewed; but that seems working as intended and I wouldn't want it to be different.

  • 1
    Thx Federico. Totally agreed, pre-publication peer reviews are a must also IMHO. But I was more (perhaps in an unclear way) hinting to the idea that the often error prone production stage could be replaced by arXiv or similar throughout the whole process. Big conferences have years ago started to label papers with "evaluated" or "reproduced" etc. Why can't we use such labels for arXiv? And "label management" would then be done by EiCs by help of reviewers if you like.
    – mfg
    Commented May 6, 2021 at 13:29
  • Review/editorial work is largely voluntary anyway. I wouldn't mind to do it equally for arXiv like for any other publisher, given I am willing to put trust in the corresponding EiC
    – mfg
    Commented May 6, 2021 at 13:34
  • The journal-ref field on Arxiv already serves a similar purpose, in the end. Commented May 6, 2021 at 13:38
  • I see, is there a particular process (on the arXiv side) associated with journal-ref that would avoid the need for going through the production stage of a publisher? Hmm, maybe we are talking of two different things here.
    – mfg
    Commented May 6, 2021 at 13:42
  • I see what you are looking for now. Many publishers are now reasonably quick to give you a DOI and put the paper online "preliminarly". And there some Arxiv "overlay journals" where the paper is published on Arxiv only. But I understand none of these answer your question exactly. Commented May 6, 2021 at 13:47

As Tim Gowers wrote in 2013 in the context of an experimental peer review website:

[...] It is easy to come up with ideas for websites where people can review papers, complete with clever protocols for how the reviewing should take place, whether it is open, reward systems, etc. etc. It’s much less easy to persuade people to use the sites that are created as a result: what is going to persuade them to make the effort, when there’s only rather a small chance that the site will become in any sense “official”?

"Becoming official" is the problem here: the publishing system's main function is no longer scientific communication, no longer improving and evaluating papers. The system has been coopted by administrators and funders, in order to manage researchers' careers via bibliometric indicators. Its continued dominance makes it difficult for alternative modes of peer review to emerge, since any new initiative must compete with an entrenched, official system.

  • 1
    Thx, @Sylvain. Let's leave aside post-peer review and stick to the best we have: pre-pub peer review. I believe in certain fields, there are reasonably well established platforms, such as arXiv or the like, that could (AFAIS) be extended to perform high-quality classical peer review and curation, just without the money-eating/encrusted structures of traditional publishers.
    – mfg
    Commented May 7, 2021 at 10:23
  • 1
    Yes, arXiv could do it, but it is not doing it and may never do it. ArXiv could have done it from the start, but did not want to scare publishers. Now it is probably too late: after "becoming official", these platforms do not evolve much. Commented May 7, 2021 at 20:55
  • When my uni made me start using the institutional Figshare a few years back, I'm sure they still had a "Comments" feature; no sign of that recently. So moving away, rather than towards. :(
    – Lou Knee
    Commented May 7, 2021 at 21:45
  • IMHO, the feature arXiv would need is to allow to label archived preprints as e.g. archived, peer-reviewed, evaluated, accepted, produced=published, etc. and make sure that those people currently voluntarily working as EiCs for journals are the only ones allowed to assign these labels exactly following their current editorial practices. I think there is a weak form of such a feature there (maybe called endorsement, don't know), but I don't really understand why it hasn't taken off.
    – mfg
    Commented May 8, 2021 at 9:36

In a sense, there is post-publication review: It just doesn't happen on websites, but in subsequent papers written by others. And the opinions of others are also recorded: A citation is like a "Like" on Facebook: The citing authors thought the cited paper offered some useful background to readers of their authors.

In other words, papers that are highly cited are, in some sense, positively post-publication reviewed. The converse is also true.

  • Very true! :) Good point! :) Commented May 6, 2021 at 21:20
  • 1
    This is not correct because people also cite papers to show that they are wrong. Commented May 6, 2021 at 23:28
  • 2
    @AnonymousPhysicist You're nitpicking. Which fraction of citations overall does this apply to, and which fraction of highly cited papers are highly cited because others are complaining about them? Commented May 7, 2021 at 5:50
  • 1
    > "people also cite papers to show that they are wrong" doesn't matter - it's still a form of review. Or, what should "post-publication review" do about bad papers - delete them completely? What should be the threshold for doing so?
    – Lou Knee
    Commented May 7, 2021 at 21:50

In a sense, post publication peer review has always been in place. Prior to publication peer review has the purpose of improving a paper before it appears (or at least prior to its formal publication). This is normally a private matter between a journal or conference and the authors.

Post publication, however, peer review has a different purpose and can no longe improve that paper. But anyone reading a paper can comment on it, make corrections to it and publish extensions or corrections. This has always been the case. Once a paper becomes visible it is open to comment by any "reviewer". This has, of course, let to some mighty intellectual battles, such as Leibniz v. Newton. If memory serves, Einstein's early papers weren't universally accepted but created some turmoil among the great names of the day. They generated a lot of comment, but also a lot of additional work attempting to refute or verify or extend them.

  • 1
    No, you are wrong. Pre publication peer review isn't about typesetting. It is about correctness and impact as well as assuring that he author properly situates the paper within the literature (citations and such). For print journals, I also suspect that typesetting is done by professionals.
    – Buffy
    Commented May 6, 2021 at 13:41
  • 1
    Then no paper can be trusted to have been vetted by experts. I can publish on my own website, actually. If you want to trust me you have to do it on your own.
    – Buffy
    Commented May 6, 2021 at 13:49
  • 1
    But the publisher need their own assurance of quality and uses reviewers they trust.
    – Buffy
    Commented May 6, 2021 at 14:49
  • 2
    One could perhaps consider citations as post-publication review.
    – Jon Custer
    Commented May 6, 2021 at 15:44
  • 6
    @Mario: You seem to be missing that at least 95% of the point of "publication" is just being able to write "accepted by J. Prestigious Results" on your CV. Everything around copyediting, production, printing, etc. is basically unimportant. Commented May 6, 2021 at 16:55

Because post-publication peer reviews are not currently recognized as useful contributions, so why do them when I could do something that would help me get a job?

  • 1
    Note that this is a rhetorical question, not exactly me feelings on the issue Commented May 6, 2021 at 23:06
  • 3
    This is a correct answer, but I'd add that conventional peer review is also very unlikely to get you a job. Commented May 6, 2021 at 23:28
  • @Anon True enough, but it is something people list under service, at least in my field Commented May 7, 2021 at 3:22

As in the other answers, the premise of the question is a bit inaccurate.

For work that turns out to be important, people look at it critically (with interest) long after it has supposedly been "refereed" and "published" (in the sense of being endorsed by a journal). And will look at it critically (with interest) as soon as it's available (e.g., online), whether or not it's submitted to a journal.

And, as has been true for 20+ years, there is the internet. Public availability of documents is no longer monopolized by "publishers" in the traditional sense. Of course, some people are less shy than others about putting online dubious stuff.

If one wants to be confident of the correctness of a journal document, one should probably check it in detail oneself, unless it is a very high profile case. In recent years, the guidance I've gotten as referee is that it is not my duty to check for correctness! And, well, in cases where no one cares much whether the conclusion is correct or not... ?!?

In math, in the U.S., we are still in the situation that, in most places, the only official way to score career/status/tenure/promotion points is by publication in refereed journals. People scrambling for tenure are not going to throw away their professional currency. And, in my experience, many people who've succeeded in this system are loathe to "make it easier" for the next generation. (I myself have recovered from any affection for artificial suffering... :)

  • Thx for these valuable insights. I am aware that step changes like I suggest won't happen without loss. But still, I think, the way how established publishers work is prehistoric. There are platforms like PLOS or f1000research which however are just not AFAIK really established in my domain.
    – mfg
    Commented May 6, 2021 at 20:43
  • This does not answer the question because it's not peer review if the outcome is not public. Commented May 6, 2021 at 23:26
  • In recent years, the guidance I've gotten as referee is that it is not my duty to check for correctness! Sounds surprising (and ridiculous, and sad). Could you elaborate a bit? Commented May 7, 2021 at 17:59
  • 4
    @RichardHardy, yes, it is surprising, since we might think exactly the opposite. Nevertheless, in the last 10-20 years, quite a few substantial journals' official statements of what the referee's task is have said that it is not the referee's job to determine correctness. Commented May 7, 2021 at 18:08

I'm taking the definition as provided in OP:

Why isn't full-fledged self-archival with classical peer review and curation, post-publication commenting, and avoidance of old-school production procedures finally taking off?

The first part has already "taken off" and is standard in my field (physics). The latter part, post-publication commenting, has not. The reason is, why should it have taken of? For something to supplant the original, it needs to be better. Post-publication commenting is not better for several reasons:

  • What would be the point? Pre-publication review can conceivably lead to the paper being improved. Post-publication review cannot.
  • What are you going to say? You can't provide suggestions for improvement because the paper is already published. You could say "This paper is great, I enjoyed reading it" which is nice to read but doesn't lead to anything. Meanwhile "This paper is bad, it should never have been accepted in the first place" is painful for the authors, especially if the criticism is public, and especially if the criticism is anonymous. Example I am aware of.
  • What would one gain from making the post-publication comment? Think about why people do peer review in the first place, and compare that to whether people should do post-publication commenting. Many of the reasons simply do not transfer: the results are already published so you don't get to read new exciting stuff before it's published, these things aren't organized by journals so you gain nothing from that front, and you can't say you're an acknowledged expert because odds are anyone who wants to submit a comment can do so.
  • Finally, it's not a reciprocal relationship. Traditional peer review is reciprocal - you peer review for others, and in return they will peer review for you. The same does not apply for post-publication commenting.

Everyone knows the traditional peer review model is flawed, but coming up with a better alternative is pretty damn hard. Post-publication commenting is an example. You might be interested in others - off the top of my head there is open peer review, where the identity of reviewers is public, and the accept-then-review model where the journal accepts every paper before putting them up for review.

  • 3
    But even if the paper can't be improved in response to the comments, the comments themselves could constitute valuable additions/improvements to the paper. Just like comments here on SE often add value to an answer even if they aren't edited into the answer. There's only so much that pre-publication peer review can do, the 100 post-publication readers will always find some unclear bits that the two per-publication reviewers found perfectly clear. If the authors could easily and publicly respond to requests for clarifications, most papers would get much easier to digest.
    – TooTea
    Commented May 7, 2021 at 12:03

Post publication peer review has a model that has proven to work, in both principle and in practice: Stack Exchange. The "research question" is generally posed by someone other than the author; there are multiple short "articles" answering the research question; peer review consists of comments and, in many cases, other articles under the same question; and, ultimately, a quantitative rating is given to the quality of the question and most of the articles, with the original questioner getting a binary (checkmark) say as the "editor."

Imagine if, for example, someone posted a question on a (relevant subject) exchange to the effect of: "Is the linked article good?" Then, assuming it didn't get taken down for being off-topic or overly vague, the resulting exchange and ratings would constitute PPPR by almost any definition. (Wikipedia would be another example, but it doesn't allow original research. You might say that little on here consists of original research, but that's really only true for the quantitative, empirical sciences, and even then a lot of published journal articles just aggregate known information.)

Most of the reasons for PPPR's failure to launch, as given in other answers, are refuted by Stack Exchange. This community consists of thousands of people, including "real" experts, who are willing to read and respond to what frequently are indisputably boring or obvious questions, for free, in an unofficial forum. Some even perform basic copyediting! True, the questions and answers, individually, are shorter than an article. But Wikipedia shows that people are also willing to read and contribute to much longer articles. And sure, there's little incentive for authors to submit their work to an "unofficial" site, but as the above example shows, you don't need the author to submit it themselves.

So, either Stack Exchange has uniquely captured lightning in a bottle (in which case it should establish its own PPPR forum) or PPPR sites have failed to copy a remarkably effective format. And that's the answer to your question.

  • why did we preserve so many art masterpieces from the past? because we did PPPR of all the crap!
    – EarlGrey
    Commented Dec 7, 2021 at 9:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .