17

I know, this is a bit of a broad question. But looking at the developments and having experienced a reasonable amount of pain with established publishers and their either complex, slow, or restrictive processes/production stages, I'm tempted to ask it here and look what others think or know about that.

Post-publication peer review has been discussed quite a while ago here and here, it's been promoted and practised on platforms such as Publons and PubPeer.

But is this idea, launched many years ago to complement "classical" peer review, really successful? Are established processes/practises being replaced/improved?

Note for clarification: I envisage something like a combination of classical pre- and an optional post-publication peer review strongly based on open self-archival platforms such as arXiv. Nothing on the academic side would change, editors and reviewers would move over to such platforms to continue their work just like before, largely unpaid. Even cult-like reputation-building, gate-keeping, etc. is still possible even if not desirable. But in domains such as CS, physics, or math, where production is largely done by authors anyway, costly, error-prone, after-acceptance production stages would be replaced by just self-archival and EiC-driven quality labelling systems as they have been used for years at top-tier conferences to certify such things as reproducibility. I think, it's time for a change. (I wonder whether this question is more something for meta.)

Further note: After the many useful comments and answers, I'm extending or rephrasing my question to: Why isn't full-fledged self-archival with classical peer review and curation, post-publication commenting, and avoidance of old-school production procedures finally taking off?

3
  • 1
    Editors and reviewers are unpaid, but the editor's secretary, who keeps track of who all the reviewers are and whether they need a reminder or not and tracks down the reviewer who changed jobs but forgot to tell the editor their new e-mail address (software can help with this but can't do all of it) has to be paid. May 6 at 21:54
  • One needs a mechanism to encourage such reviews to be collected. For example, all referees/reviewers for awards and promotions could be requested to review at least one paper in order for their letter to carry additional weight.
    – Kapil
    May 7 at 4:46
  • 1
    We need a robust reddit-style comment system on the Arxiv so that people can discuss papers easily and openly. The comments will be very enlightening. The comment system should support upvoting and downvoting comments and should support mathjax. This needs to exist.
    – littleO
    May 8 at 2:56
28

Most papers that are published are uninteresting.

The median paper has less than one reader not counting the authors and the reviewers.

The current reviewing system relies on a network of responsibilities that are independent of the author. The editors have a responsibility to the community as a whole to get papers reviewed properly, and the community gives them esteem for performing this duty. The reviewers have a responsibility to the editor to actually do reviews, and the editor is usually some influential person (and in any case will probably at some point be the editor for one of the reviewer's papers).

Suppose I have written one of my usual mediocre papers that is among the 70% (my estimate) of papers that are never read. How would post-publication peer review be organized for it? If I have to find reviewers myself, then everyone is going to say 'No' or do a bad job, because no one feels any need to impress me and it's not a very interesting paper to read. Moreover, the best reviewers for my paper are people with a little higher status than me, and they would definitely say 'No' to me, but they might say 'Yes' to an editor. (Keep in mind that, as the system is set up, some top people in a field do far more reviewing than they write papers, while many people write one write-only paper every several years (or grad students who stop doing research write one paper from their dissertation) and never review.)

10
  • 4
    Yup! Papers are awful and refereeing is the worst. There is one other motivating factor here that you don’t mention, I referee young mathematicians work (even if I don’t feel obligated to the editor) because I want them to get jobs. But that mostly applies just to postdocs. May 6 at 15:57
  • 3
    Note that post-publication (or non-publication) peer review happens pretty well for interesting papers that people want to read. Perelman’s work went through a very extensive process to verify the work and expand on the details despite never being published. Conversely, everyone knows where the hole is in Mochizuki’s work from outside readers, even though it’s published and the referees didn’t do their job. May 6 at 16:03
  • 12
    Actually, it isn't that most papers are uninteresting. It is that the audience for extremely specialized work is limited in many fields.
    – Buffy
    May 6 at 21:53
  • 6
    Do you have a source for The median paper has less than one reader not counting the authors and the reviewers? I can ask a separate question on this if you consider it off-topic for this one.
    – Allure
    May 7 at 6:10
  • 1
    @SylvainRibault: Because research is not the only purpose of research. I know someone with a PhD who got their bachelor's degree from Alma College. There, the professors teach 360 classroom hours or more a year, leaving little time for research. Alma College and the mathematical community would like to know that the professors there have a good sense of what mathematical research is so that we can trust the recommendation letters they write (and for other purposes). So we want to review their occasional papers even though they don't seriously advance mathematics. May 7 at 21:14
16

In a sense, it has taken off; you just need to change your definitions: in mathematics, most papers get peer reviewed after they have appeared on the Arxiv and are available to the public.

Results then appear in prestigious journals, so they are ultimately trusted and used to evaluate researchers and compare academic egos, only after they have been carefully been reviewed; but that seems working as intended and I wouldn't want it to be different.

11
  • 1
    Thx Federico. Totally agreed, pre-publication peer reviews are a must also IMHO. But I was more (perhaps in an unclear way) hinting to the idea that the often error prone production stage could be replaced by arXiv or similar throughout the whole process. Big conferences have years ago started to label papers with "evaluated" or "reproduced" etc. Why can't we use such labels for arXiv? And "label management" would then be done by EiCs by help of reviewers if you like.
    – Mario
    May 6 at 13:29
  • Review/editorial work is largely voluntary anyway. I wouldn't mind to do it equally for arXiv like for any other publisher, given I am willing to put trust in the corresponding EiC
    – Mario
    May 6 at 13:34
  • The journal-ref field on Arxiv already serves a similar purpose, in the end. May 6 at 13:38
  • I see, is there a particular process (on the arXiv side) associated with journal-ref that would avoid the need for going through the production stage of a publisher? Hmm, maybe we are talking of two different things here.
    – Mario
    May 6 at 13:42
  • I see what you are looking for now. Many publishers are now reasonably quick to give you a DOI and put the paper online "preliminarly". And there some Arxiv "overlay journals" where the paper is published on Arxiv only. But I understand none of these answer your question exactly. May 6 at 13:47
5

In a sense, post publication peer review has always been in place. Prior to publication peer review has the purpose of improving a paper before it appears (or at least prior to its formal publication). This is normally a private matter between a journal or conference and the authors.

Post publication, however, peer review has a different purpose and can no longe improve that paper. But anyone reading a paper can comment on it, make corrections to it and publish extensions or corrections. This has always been the case. Once a paper becomes visible it is open to comment by any "reviewer". This has, of course, let to some mighty intellectual battles, such as Leibniz v. Newton. If memory serves, Einstein's early papers weren't universally accepted but created some turmoil among the great names of the day. They generated a lot of comment, but also a lot of additional work attempting to refute or verify or extend them.

12
  • 1
    No, you are wrong. Pre publication peer review isn't about typesetting. It is about correctness and impact as well as assuring that he author properly situates the paper within the literature (citations and such). For print journals, I also suspect that typesetting is done by professionals.
    – Buffy
    May 6 at 13:41
  • 1
    Then no paper can be trusted to have been vetted by experts. I can publish on my own website, actually. If you want to trust me you have to do it on your own.
    – Buffy
    May 6 at 13:49
  • 1
    But the publisher need their own assurance of quality and uses reviewers they trust.
    – Buffy
    May 6 at 14:49
  • 2
    One could perhaps consider citations as post-publication review.
    – Jon Custer
    May 6 at 15:44
  • 6
    @Mario: You seem to be missing that at least 95% of the point of "publication" is just being able to write "accepted by J. Prestigious Results" on your CV. Everything around copyediting, production, printing, etc. is basically unimportant. May 6 at 16:55
5

Because post-publication peer reviews are not currently recognized as useful contributions, so why do them when I could do something that would help me get a job?

3
  • 1
    Note that this is a rhetorical question, not exactly me feelings on the issue May 6 at 23:06
  • 2
    This is a correct answer, but I'd add that conventional peer review is also very unlikely to get you a job. May 6 at 23:28
  • @Anon True enough, but it is something people list under service, at least in my field May 7 at 3:22
5

As in the other answers, the premise of the question is a bit inaccurate.

For work that turns out to be important, people look at it critically (with interest) long after it has supposedly been "refereed" and "published" (in the sense of being endorsed by a journal). And will look at it critically (with interest) as soon as it's available (e.g., online), whether or not it's submitted to a journal.

And, as has been true for 20+ years, there is the internet. Public availability of documents is no longer monopolized by "publishers" in the traditional sense. Of course, some people are less shy than others about putting online dubious stuff.

If one wants to be confident of the correctness of a journal document, one should probably check it in detail oneself, unless it is a very high profile case. In recent years, the guidance I've gotten as referee is that it is not my duty to check for correctness! And, well, in cases where no one cares much whether the conclusion is correct or not... ?!?

In math, in the U.S., we are still in the situation that, in most places, the only official way to score career/status/tenure/promotion points is by publication in refereed journals. People scrambling for tenure are not going to throw away their professional currency. And, in my experience, many people who've succeeded in this system are loathe to "make it easier" for the next generation. (I myself have recovered from any affection for artificial suffering... :)

4
  • Thx for these valuable insights. I am aware that step changes like I suggest won't happen without loss. But still, I think, the way how established publishers work is prehistoric. There are platforms like PLOS or f1000research which however are just not AFAIK really established in my domain.
    – Mario
    May 6 at 20:43
  • This does not answer the question because it's not peer review if the outcome is not public. May 6 at 23:26
  • In recent years, the guidance I've gotten as referee is that it is not my duty to check for correctness! Sounds surprising (and ridiculous, and sad). Could you elaborate a bit? May 7 at 17:59
  • 3
    @RichardHardy, yes, it is surprising, since we might think exactly the opposite. Nevertheless, in the last 10-20 years, quite a few substantial journals' official statements of what the referee's task is have said that it is not the referee's job to determine correctness. May 7 at 18:08
5

I'm taking the definition as provided in OP:

Why isn't full-fledged self-archival with classical peer review and curation, post-publication commenting, and avoidance of old-school production procedures finally taking off?

The first part has already "taken off" and is standard in my field (physics). The latter part, post-publication commenting, has not. The reason is, why should it have taken of? For something to supplant the original, it needs to be better. Post-publication commenting is not better for several reasons:

  • What would be the point? Pre-publication review can conceivably lead to the paper being improved. Post-publication review cannot.
  • What are you going to say? You can't provide suggestions for improvement because the paper is already published. You could say "This paper is great, I enjoyed reading it" which is nice to read but doesn't lead to anything. Meanwhile "This paper is bad, it should never have been accepted in the first place" is painful for the authors, especially if the criticism is public, and especially if the criticism is anonymous. Example I am aware of.
  • What would one gain from making the post-publication comment? Think about why people do peer review in the first place, and compare that to whether people should do post-publication commenting. Many of the reasons simply do not transfer: the results are already published so you don't get to read new exciting stuff before it's published, these things aren't organized by journals so you gain nothing from that front, and you can't say you're an acknowledged expert because odds are anyone who wants to submit a comment can do so.
  • Finally, it's not a reciprocal relationship. Traditional peer review is reciprocal - you peer review for others, and in return they will peer review for you. The same does not apply for post-publication commenting.

Everyone knows the traditional peer review model is flawed, but coming up with a better alternative is pretty damn hard. Post-publication commenting is an example. You might be interested in others - off the top of my head there is open peer review, where the identity of reviewers is public, and the accept-then-review model where the journal accepts every paper before putting them up for review.

1
  • 2
    But even if the paper can't be improved in response to the comments, the comments themselves could constitute valuable additions/improvements to the paper. Just like comments here on SE often add value to an answer even if they aren't edited into the answer. There's only so much that pre-publication peer review can do, the 100 post-publication readers will always find some unclear bits that the two per-publication reviewers found perfectly clear. If the authors could easily and publicly respond to requests for clarifications, most papers would get much easier to digest.
    – TooTea
    May 7 at 12:03
5

As Tim Gowers wrote in 2013 in the context of an experimental peer review website:

[...] It is easy to come up with ideas for websites where people can review papers, complete with clever protocols for how the reviewing should take place, whether it is open, reward systems, etc. etc. It’s much less easy to persuade people to use the sites that are created as a result: what is going to persuade them to make the effort, when there’s only rather a small chance that the site will become in any sense “official”?

"Becoming official" is the problem here: the publishing system's main function is no longer scientific communication, no longer improving and evaluating papers. The system has been coopted by administrators and funders, in order to manage researchers' careers via bibliometric indicators. Its continued dominance makes it difficult for alternative modes of peer review to emerge, since any new initiative must compete with an entrenched, official system.

4
  • 1
    Thx, @Sylvain. Let's leave aside post-peer review and stick to the best we have: pre-pub peer review. I believe in certain fields, there are reasonably well established platforms, such as arXiv or the like, that could (AFAIS) be extended to perform high-quality classical peer review and curation, just without the money-eating/encrusted structures of traditional publishers.
    – Mario
    May 7 at 10:23
  • 1
    Yes, arXiv could do it, but it is not doing it and may never do it. ArXiv could have done it from the start, but did not want to scare publishers. Now it is probably too late: after "becoming official", these platforms do not evolve much. May 7 at 20:55
  • When my uni made me start using the institutional Figshare a few years back, I'm sure they still had a "Comments" feature; no sign of that recently. So moving away, rather than towards. :(
    – Lou Knee
    May 7 at 21:45
  • IMHO, the feature arXiv would need is to allow to label archived preprints as e.g. archived, peer-reviewed, evaluated, accepted, produced=published, etc. and make sure that those people currently voluntarily working as EiCs for journals are the only ones allowed to assign these labels exactly following their current editorial practices. I think there is a weak form of such a feature there (maybe called endorsement, don't know), but I don't really understand why it hasn't taken off.
    – Mario
    May 8 at 9:36
4

In a sense, there is post-publication review: It just doesn't happen on websites, but in subsequent papers written by others. And the opinions of others are also recorded: In a sense, a citation is like a "Like" on Facebook: The citing authors thought the cited paper offered some useful background to readers of their authors.

In other words, papers that are highly cited are, in a sense, positively post-publication reviewed. The converse is also true.

4
  • Very true! :) Good point! :) May 6 at 21:20
  • 1
    This is not correct because people also cite papers to show that they are wrong. May 6 at 23:28
  • 1
    @AnonymousPhysicist You're nitpicking. Which fraction of citations overall does this apply to, and which fraction of highly cited papers are highly cited because others are complaining about them? May 7 at 5:50
  • > "people also cite papers to show that they are wrong" doesn't matter - it's still a form of review. Or, what should "post-publication review" do about bad papers - delete them completely? What should be the threshold for doing so?
    – Lou Knee
    May 7 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.