3

My understanding is that publishing model of academic research (i.e. publishing in journals) is a bit broken mostly in the sense that non-significant results are not published.

Key factors are:

  • Journals have a monetary incentive to publish 'exciting' results
  • Researchers can be unwilling to share non-significant results early for fear of having credit for their idea stolen.

This seems to be a bit at odds with the scientific method where by results of a experiment should be reproducible, but if non-significant results are being ignored then, then many published results may just be a p-value hack (obligatory xkcd). This does seem to be a real problem - ie. The Replication Crisis.

On the other hand, I'm very impressed with the state of open source software - particularly within web development, (it seems that) the vast majority of modern web dev tooling is created openly and transparently via the open source model.

The question is - are there attempts to develop an alternative publishing model for scientific experimentation?

Key features I'm particularly interested in, is the ability to express an experiment in a machine readable format, and have similar experiments be discoverable.

6
  • 1
    I think RSOS is an example. royalsocietypublishing.org/rsos/replication-studies
    – Neuchâtel
    Commented Jan 4, 2023 at 6:02
  • "ability to express an experiment in a machine readable format" - how?! I work on somewhat related things for a number of years now, and this proposal sounds waaaaay overly ambitious. paperswithcode exist in the field of ML, but it is just a collection of github repos - running the code is a massive, hard-to-automate effort. Or did you mean something like Zenodo, just with more granular tags?
    – Lodinn
    Commented Jan 4, 2023 at 16:43
  • 3
    "more open source" and "like inclusion of non-significant results" don't really seem compatible to me - the latter is not an example of "open source".
    – Bryan Krause
    Commented Jan 4, 2023 at 17:43
  • 1
    @BryanKrause This! One could actually argue very very many non-significant results are constantly being published (and not necessarily in bad journals). I fail to see what this has to do with reproducibility... Commented Jan 4, 2023 at 22:55
  • @ZeroTheHero Eh, not sure I follow you, either; pretty much every meta analysis ever performed shows evidence of publication bias - that is, evidence that studies that return p-values that are not significant are not published as often as studies that return p-values that are significant, even though one would expect a mixture of both if effects were real (but small), as expected for most tested hypotheses. The issue with reproducibility is that for effects that are actually zero, this publication bias means that the spurious non-zero results are published and contradictory ones are not.
    – Bryan Krause
    Commented Jan 4, 2023 at 23:01

3 Answers 3

3

The biology journal eLife, which is regarded as a very good journal, is experimenting with a model where papers are published alongside the peer reviews, irrespective of what the reviews say. This should allow the publication of results which the reviewers don't view as novel or significant. https://elifesciences.org/about

Once upon a time they talked about publishing executable articles, but I don't see anything about that after a quick glance at the website, so perhaps that experiment wasn't a success.

UPDATE:

This was just announced https://elifesciences.org/labs/dc5acbde/welcome-to-a-new-era-of-reproducible-publishing

The idea is to make the paper a sort of notebook, comparable to Jupiter or RMarkdown, allowing the reader direct access to the code and data, allowing the change and reexecute the code etc. I assume is runs on some sort of firewalled, managed VM system, like mybinder.org

4
  • 2
    The possibility of malware imbedded in executables probably discouraged "executable articles". Commented Jan 4, 2023 at 17:32
  • I think the idea was to make use of infrastructure simlar to that of mybinder.org to run computational notebooks in non-connected virtual machines. But to be honest, I suspect that was too limiting to be useful. Commented Jan 4, 2023 at 19:03
  • This just announced elifesciences.org/labs/dc5acbde/… Commented Jan 14, 2023 at 20:52
  • @IanSudbury, probably you should incorporate these update-comments into your answer, so that they don't get lost if/when comments are purged? Commented Jan 14, 2023 at 22:03
2

With some clarifications obtained from the comments, I would now attempt to answer. Brace yourself for a long and, perhaps, a bit controversial post.

Some efforts are made to make science more reproducible and transparent. But I believe that what you are hoping to achieve here is fundamentally incompatible with science as we know it. And the key reason for that is that (applied) science is not the most rigorous activity on the face of the planet. Perhaps it is a bit ironic given that it is used as a standard of rigor and thoroughness, but the industry might actually have an edge over it.

Scientists' job is to find new things that might be useful. It is unrealistic to expect them to be able to make things ISO-compliant and provide installation instructions for half a dozen of distros and fix every bug arising from an update in an external dependency. Some fields do better in this regard, some worse, but without serious industry backing, the resources required for this level of coverage and continuous support are just not there. The same argument can be made for open source software. The way the whole system is currently set up is that scientists do curation and heuristics, and only then some serious engineering effort is poured in. I would expect the bottom line to rise over time, but still be consistently behind that in the industry. It is, and will be for foreseeable future, a tooling problem: most scientific software can produce, say, r^2 values, but that would be far behind what is actually required. As soon as you know the requirements and specification in enough detail, it is not science, it is engineering, and not the most creative part of the latter, either. We will not have nearly enough people and money to do bleeding edge research with full reproducibility - instead, the best studies will eventually stand out and grow, while the worse ones will be buried under the sands of time. Like in many things in life, it is often more efficient to invest $1000 in one idea than $1 in a 1000 ideas each. And the balance is generally somewhere in between, some ideas will get $500, some $100, some $1. Scientists generate lots of ideas, so naturally, most of them would not be getting $1 in this example just to keep going. What that $1 gets you changes over time, but it is never much.

As an anecdote, I once sat down with people from a big IT company who were giving us a product presentation for a joint project and describing roles for a team of 7-10 people who were supposedly doing the technical part of this project. That felt utterly out of touch, I could feel schizophrenia kicking in, as I was literally the only person covering all these roles on our side. And every time I configure iptables or Docker containers or figure out JWT authorization, I am hurting my research career (and, arguably, wasting taxpayers' money). There are better examples of academia and industry working together: if you look at some of the foundational works in machine learning or some branches of computer science, the code is very poorly written. There are no automatic tests - this is another huge problem, by the way: to write tests, you need to know the answer already. Early internet name resolution system in a form of a plain text file was maintained by just one amazing gal before someone finally decided to invest resources for a proper overhaul. All these things were solved, just not by the academia alone.

Another problem to consider is a socioeconomic one. You have citizens paying taxes, and they are ultimately interested in consumer goods and well-being of the industry. Being an average Joe who can't do much of rocket science but, you know, comes to the factory five days a week and produces parts for Quantum Superiority™ 3000. Prestige is fine and well, but the reason many people care about it is not having to pay triple for Quantum Leap™ 2 produces in some other country because they do not have a comparable technology. Scientists are generally more liberal and cooperative people than average, but for a government, 100% reproducibility with minimal effort in all but the most fundamental sciences means securing international cooperation and playing the centipede game on a global scale. Scientists may strive to benefit all of the humanity, but funding agencies operate differently, and there are lots of politics involved in these decisions (say, NSF would strive to reduce the friction and waste domestically while benefitting scientists from Russia and China as little as possible).

You list web dev tooling as an example, but it only became open and any good at all when companies started competing on something else. At first, the concept of the wheel is a closely guarded secret, but then it is out, you are building cars, and finally may agree to come together and standardize wheels and roads.

All that is to say that we are not getting zero-effort full reproducibility anytime soon, if ever. One could define some arbitrary standards and develop tooling to easily achieve them, but that would just move the goalposts. Science keeps self-correcting to stay one hit ahead of the flock, and paradoxically that means striving for making our lives just hard enough to stay busy. As soon as something "just works", we would move on to the next thing that still does not. This is just our human condition.

0

People absolutely try. I can't name the journals that try these off the top of my head, but as you can see from that Wikipedia section, there are journals that:

  • Do double-blind peer review (authors don't know who the reviewers are, and vice versa)
  • Do triple-blind peer review (authors & editors & reviewers don't know who each other are)
  • Do open peer review (everyone knows who everyone else is)
  • Do open peer reports (reviews are published together with the paper)
  • Do open participation (reviewers self-select to review the paper)
  • Do post-publication peer review (every paper is published, reviews are done after publication)
  • Do results-blind peer review (reviewers receive a manuscript where the results & conclusions are omitted)
  • Do two-stage results-blind peer review (review done in two stages; in the first stage reviewers don't know the results/conclusions, in the second stage they do)
  • Do novelty-blind peer review (reviewers are specifically instructed not to comment on whether the paper is novel, only if it is correct)

The fact that the traditional model has endured is a sign of how robust it is. Everyone knows it is flawed, but nobody has been able to come up with a better model.

2
  • 1
    Which of these are relevant in encouraging publication of "non-significant results"? Isn't the OP mostly about that?
    – GEdgar
    Commented Jan 5, 2023 at 10:49
  • @GEdgar Do results-blind peer review (reviewers receive a manuscript where the results & conclusions are omitted) <--
    – Allure
    Commented Jan 5, 2023 at 10:53

Not the answer you're looking for? Browse other questions tagged .