16

In order to publish a research paper in a journal, one of the requirement may be the reproducibility of results.

I'm wondering, in academia, how does an un-reproducible result even exist?

Is there any classical example for it, in any branch of academia research?

  • 2
    What kind of reproducibility are your referring to? Theoretical reproducibility or practical reproducibility? If your experiment requires a large hadron collider or other pretty unique equipment or circumstances, the former might be achieved by careful method description but the latter is almost impossible. However, there are many papers that don't provide sufficient details to reproduce the experiment even in theory. And many attempts to actually reproduce results from papers that do provide them have failed. There is ongoing discussion about a reproducibility crisis ... – Roland Aug 24 at 13:48
  • 10
    Please define what you mean by "reproducible," people have different understandings – Azor Ahai -- he him Aug 24 at 13:58
  • 16
    See "Journal of Irreproducible Results" ;-) – Captain Emacs Aug 24 at 14:37
  • 1
    en.m.wikipedia.org/wiki/… – UJM Aug 24 at 15:25
  • 8
    I'm reminded of Adam Ruins Everything: "One research team just replicated 100 famous psychology studies, but found they couldn't reproduce about 60% of them... Another revisited 67 major drug studies, and found that about 75% didn't match their results... Another team zeroed in on 53 recent cancer studies, and couldn't reproduce 47 of them... One estimate suggests that, in the U.S., we spend $28 billion a year on biomedical research that can't be reproduced." – Laurel Aug 24 at 23:44
40

It's important to define what reproducibility actually means and in what context it is used. Science deals with things that can be reproduced in principle: If you managed to re-create the exact same situation, you would be able to get the same result.

But in practice, that doesn't always mean that you can re-create the situation: You might have measured the seismic waves of a very large earthquake in Indonesia. Or you might have seen photons of a nearby supernova. Neither of these conditions can be created by humans, and so the experiment can not be repeated in practice, though in principle it could be. A related situation happens if it is impractical to do so: If the original experiment was done with a ten-billion $ machine (say, a particle accelerator, a nuclear fusion reaction), then yes you could repeat the experiment, but you probably find yourself in financial trouble if you tried. There are also valid research results that should not be reproduced, even if they could: Say, whatever we may have learned from the Tuskeegee syphilis study or the Stanford prison experiment might be scientifically correct, even repeatable, but one can only hope that nobody will ever try to repeat these studies.

Finally, there are often practical constraints: If you take a picture of turbulence in a pipe, you will not be able to recreate the same picture because turbulence is a chaotic process; similarly, if you try to do experiments on a single cell and count the number of molecules of a specific kind, you're likely going to find that it depends sensitively on temperature, time of day, etc. That doesn't mean that the science is wrong: In both cases, statistical assessments of the results may still be valid, even if you can't recreate the specific numbers.

Of course, there are also experiments that really can't be reproduced: Someone published the results of an experiment that seemed reasonable to them and to the reviewers, but the measuring device had a mechanical defect and consequently every number in the publication is just wrong and the measured effect does not actually exist. This of course shouldn't happen, but it does happen in practice. There are also common statistical problems in studies that involve a small number of human subjects where the random, involuntary, or voluntary choice of subjects suggested an effect that, if repeated on a larger and more random cohort does not actually exist.

| improve this answer | |
  • 4
    This answer could be improved by expanding a bit on when results are unreproducible (i.e. what the question asks). A mechanical defect isn't the only reason results might be unreproducible. I would imagine it's also one of the less likely cause of unreproducibility. Also, where does OP "confuse what reproducibility actually means"? I can't see that anywhere. In the question there are mostly just questions asking what it means without any assertions or implications about what it means. – NotThatGuy Aug 25 at 11:22
  • 1
    @uhoh Fair point -- how's this? – Wolfgang Bangerth Aug 25 at 13:33
  • Looks great, thanks! – uhoh Aug 25 at 13:35
  • 1
    Another possibility leading to a result being unreproducible is insufficient documentation, logging, or provenance and version tracking. – Jake Aug 25 at 15:42
  • 1
    Also, a fabricated result would be unreproducible, and in some cases "unreproducible" may be a euphemism for fraudulent. – Jake Aug 25 at 15:44
16

Wikipedia has an excellent article on the current Replication Crisis (or Reproducibility Crisis), and I can't hope to improve on it. I would recommend that you start by reading that article, the many examples included, and the other referenced links therein.

The replication crisis (or replicability crisis or reproducibility crisis) is, as of 2020, an ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. The replication crisis affects the social sciences and medicine most severely. The crisis has long-standing roots; the phrase was coined in the early 2010s as part of a growing awareness of the problem. The replication crisis represents an important body of research in the field of metascience...

Glenn Begley and John Ioannidis proposed these causes:

  • Generation of new data/publications at an unprecedented rate.
  • Majority of these discoveries will not stand the test of time.
  • Failure to adhere to good scientific practice and the desperation to publish or perish.
  • Multiple varied stakeholders

They conclude that no party is solely responsible, and no single solution will suffice. In fact, some predictions of an impending crisis in the quality control mechanism of science can be traced back several decades...

Philosopher and historian of science Jerome R. Ravetz predicted in his 1971 book Scientific Knowledge and Its Social Problems that science – in its progression from "little" science composed of isolated communities of researchers, to "big" science or "techno-science" – would suffer major problems in its internal system of quality control. Ravetz recognized that the incentive structure for modern scientists could become dysfunctional, now known as the present 'publish or perish' challenge, creating perverse incentives to publish any findings, however dubious. According to Ravetz, quality in science is maintained only when there is a community of scholars linked by a set of shared norms and standards, all of whom are willing and able to hold one another accountable.

Keep in mind that modern scientific studies have a probabilistic aspect in how their random sample has been determined. It's the goal of statistical inference to express and clarify this fact. For example: Say there's a deadly disease, a researcher creates a medication that in truth does nothing, but just by fortune happens to give it to the only 10 people in the world who will recover on their own. Then that study will appear to be an amazing success and surely get published, even though no one will ever be able to recreate the effect. This is called Publication Bias and with more and more scientific studies done over time, it's theorized that a majority of published papers may now be in this category.

| improve this answer | |
  • 2
    I was about to post an answer that focuses on too small or non-representative samples (esp. in social sciences) as well as p-hacking & related issues when I realized that all this is covered in the article you cite.Maybe you could summarize/highlight some of these points directly in your answer because I think these issues receive less attention than they should.Aspects covered in other answers are important (cost of reproducing results, one-off experiments like a supernova, computational power, lack of information, etc), but IMO these statistical aspects are key for (non-)reproducibility. – CL. Aug 25 at 14:06
  • 1
    @CL.: I agree the statistical aspects are key. Initially I avoided getting into that to keep the answer brief, and because in the linked article the statistical discussion is field-dependent. I added a final paragraph trying to very briefly scope out the issue, thanks for the suggestion. – Daniel R. Collins Aug 25 at 14:54
10

Enough information to reproduce

If we look at a journal requirement for the results to be reproducible, it's not about the result being reproducible in principle, but about the possibility to reproduce it based on the contents of your paper.

For example, your paper may describe a result that you can reproduce but that is effectively not reproducible by omitting key details of the experimental setup or by relying on data that's classified or otherwise unavailable to others. If so, you're effectively asking the journal, reviewers and the wider community (who might actually attempt to reproduce the results later) to accept your results as true based on pure faith and goodwill - and they may refuse to do so. This limitation generally is not strictly applied to things that are very difficult or very expensive to reproduce (e.g. you need the equivalent of the Large Hadron Collider to replicate LHC results, but physics still wants to publish results coming out of LHC even if noone else has a comparable particle accelerator), but for situations where practical reproduction is plausible, it makes sense to mandate that the authors include in the paper the information required to do so.

| improve this answer | |
  • I think this is the best answer, even if it does not technically answer the question as asked – Vincent Aug 25 at 17:20
7

Consider data mining or machine learning research. Normally, you invent a new algorithm, run it against state-of-the-art competitors on publicly available datasets, and if your algorithm performs significantly better, then you have a paper. Assuming that the paper clearly describes the algorithm and the experimental setup, these results are reproducible: if anyone were to repeat the steps, they should arrive at the same conclusion.

Nowadays, massive IT companies with research departments submit research papers to data mining and machine learning conferences as well. Imagine that Google invents a new algorithm, runs it against state-of-the-art competitors on publicly available datasets, and reports a significantly better performance in their paper. Sounds good, right? However, the new algorithm might be computationally expensive to a degree that prohibits reproducibility: maybe the algorithm needs access to Google's proprietary servers in order to be fed with enough computation power to allow the algorithm to finish before the universe implodes. The paper is written in the exact same way, with enough detail such that if anyone were to repeat the steps, they should arrive at the same conclusion. However, I would argue that this is not reproducible research: the average research scientist at the average university will not be able to repeat the steps, by lack of access to computation facilities.

The tricky thing is that the research may very well be valid. Unfortunately, the average reviewer has no way of knowing this. But that's hardly Google's fault, so they should be able to publish their papers. But it's not really reproducible. So this is a very gray area.

| improve this answer | |
  • 1
    I would consider a paper that doesn't clearly describe the algorithm or the experimental setup, or where someone repeating the steps arrives at a different conclusion (as from your first paragraph) to be more clearly unreproducible than steps simply being too expensive to run for (most) others. The latter can also just be phrased as having unverifiable reproducibility. – NotThatGuy Aug 25 at 11:42
1

In order to publish a research paper in a journal, one of the requirement may be the reproducibility of results.

Your assumption is false. Only certain kinds of journals require reproducibility.

Classic examples of unreproducible results include:

  • Case studies of unique circumstances. For rare diseases, these are very valuable, but they apply to many non-medical fields of research too.
  • All observational astronomy (there is only one universe).
  • Statistical fluctuations.
  • Expensive experiments.
  • Errors.

Only the errors should not be published.

| improve this answer | |
  • 7
    There is the word "may" in the quote. It makes it strange to continue with "your assumption is false" and next verifying the quote as true. – Tommi Aug 25 at 9:41
1

To add what others have already written, reproducibility in academic publishing mainly deals with the extent of how you have described your research methodology.

UPDATE: Un-reproducible results are largely frowned upon and a result of not describing the study properly. There is a place in science for experience reports and conjectures, however these should be framed as such.

Say, you are dealing with something unique like a supernova exploding. There is no way to recreate it in any practical sense. However, you can describe the star (type, size, distance, composition, environment, details of the explosion, etc etc) to provide the reader with enough details to understand what you are studying.

Similarly, you describe what kind of tools (telescopes, detectors etc.) you have used to collect data, what kind of methods you used to analyze the data, and so on. Thus, the reader can trace back your conclusions to data and to the circumstances of how the data was collected, and to the phenomenon being investigated.

Thus, the whole study may not be reproducible but parts of it can. Like, using the same equipment in the same way, applying the same data analysis methods, apply your reasoning to draw conclusions.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.