Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It only takes a minute to sign up.
Sign up to join this community
In order to publish a research paper in a journal, one of the requirement may be the reproducibility of results.
I'm wondering, in academia, how does an un-reproducible result even exist?
Is there any classical example for it, in any branch of academia research?
It's important to define what reproducibility actually means and in what context it is used. Science deals with things that can be reproduced in principle: If you managed to re-create the exact same situation, you would be able to get the same result.
But in practice, that doesn't always mean that you can re-create the situation: You might have measured the seismic waves of a very large earthquake in Indonesia. Or you might have seen photons of a nearby supernova. Neither of these conditions can be created by humans, and so the experiment can not be repeated in practice, though in principle it could be. A related situation happens if it is impractical to do so: If the original experiment was done with a ten-billion $ machine (say, a particle accelerator, a nuclear fusion reaction), then yes you could repeat the experiment, but you probably find yourself in financial trouble if you tried. There are also valid research results that should not be reproduced, even if they could: Say, whatever we may have learned from the Tuskeegee syphilis study or the Stanford prison experiment might be scientifically correct, even repeatable, but one can only hope that nobody will ever try to repeat these studies.
Finally, there are often practical constraints: If you take a picture of turbulence in a pipe, you will not be able to recreate the same picture because turbulence is a chaotic process; similarly, if you try to do experiments on a single cell and count the number of molecules of a specific kind, you're likely going to find that it depends sensitively on temperature, time of day, etc. That doesn't mean that the science is wrong: In both cases, statistical assessments of the results may still be valid, even if you can't recreate the specific numbers.
Of course, there are also experiments that really can't be reproduced: Someone published the results of an experiment that seemed reasonable to them and to the reviewers, but the measuring device had a mechanical defect and consequently every number in the publication is just wrong and the measured effect does not actually exist. This of course shouldn't happen, but it does happen in practice. There are also common statistical problems in studies that involve a small number of human subjects where the random, involuntary, or voluntary choice of subjects suggested an effect that, if repeated on a larger and more random cohort does not actually exist.
Wikipedia has an excellent article on the current Replication Crisis (or Reproducibility Crisis), and I can't hope to improve on it. I would recommend that you start by reading that article, the many examples included, and the other referenced links therein.
The replication crisis (or replicability crisis or reproducibility crisis) is, as of 2020, an ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate or reproduce. The replication crisis affects the social sciences and medicine most severely. The crisis has long-standing roots; the phrase was coined in the early 2010s as part of a growing awareness of the problem. The replication crisis represents an important body of research in the field of metascience...
Glenn Begley and John Ioannidis proposed these causes:
- Generation of new data/publications at an unprecedented rate.
- Majority of these discoveries will not stand the test of time.
- Failure to adhere to good scientific practice and the desperation to publish or perish.
- Multiple varied stakeholders
They conclude that no party is solely responsible, and no single solution will suffice. In fact, some predictions of an impending crisis in the quality control mechanism of science can be traced back several decades...
Philosopher and historian of science Jerome R. Ravetz predicted in his 1971 book Scientific Knowledge and Its Social Problems that science – in its progression from "little" science composed of isolated communities of researchers, to "big" science or "techno-science" – would suffer major problems in its internal system of quality control. Ravetz recognized that the incentive structure for modern scientists could become dysfunctional, now known as the present 'publish or perish' challenge, creating perverse incentives to publish any findings, however dubious. According to Ravetz, quality in science is maintained only when there is a community of scholars linked by a set of shared norms and standards, all of whom are willing and able to hold one another accountable.
Keep in mind that modern scientific studies have a probabilistic aspect in how their random sample has been determined. It's the goal of statistical inference to express and clarify this fact. For example: Say there's a deadly disease, a researcher creates a medication that in truth does nothing, but just by fortune happens to give it to the only 10 people in the world who will recover on their own. Then that study will appear to be an amazing success and surely get published, even though no one will ever be able to recreate the effect. This is called Publication Bias and with more and more scientific studies done over time, it's theorized that a majority of published papers may now be in this category.
If we look at a journal requirement for the results to be reproducible, it's not about the result being reproducible in principle, but about the possibility to reproduce it based on the contents of your paper.
For example, your paper may describe a result that you can reproduce but that is effectively not reproducible by omitting key details of the experimental setup or by relying on data that's classified or otherwise unavailable to others. If so, you're effectively asking the journal, reviewers and the wider community (who might actually attempt to reproduce the results later) to accept your results as true based on pure faith and goodwill - and they may refuse to do so. This limitation generally is not strictly applied to things that are very difficult or very expensive to reproduce (e.g. you need the equivalent of the Large Hadron Collider to replicate LHC results, but physics still wants to publish results coming out of LHC even if noone else has a comparable particle accelerator), but for situations where practical reproduction is plausible, it makes sense to mandate that the authors include in the paper the information required to do so.
Consider data mining or machine learning research. Normally, you invent a new algorithm, run it against state-of-the-art competitors on publicly available datasets, and if your algorithm performs significantly better, then you have a paper. Assuming that the paper clearly describes the algorithm and the experimental setup, these results are reproducible: if anyone were to repeat the steps, they should arrive at the same conclusion.
Nowadays, massive IT companies with research departments submit research papers to data mining and machine learning conferences as well. Imagine that Google invents a new algorithm, runs it against state-of-the-art competitors on publicly available datasets, and reports a significantly better performance in their paper. Sounds good, right? However, the new algorithm might be computationally expensive to a degree that prohibits reproducibility: maybe the algorithm needs access to Google's proprietary servers in order to be fed with enough computation power to allow the algorithm to finish before the universe implodes. The paper is written in the exact same way, with enough detail such that if anyone were to repeat the steps, they should arrive at the same conclusion. However, I would argue that this is not reproducible research: the average research scientist at the average university will not be able to repeat the steps, by lack of access to computation facilities.
The tricky thing is that the research may very well be valid. Unfortunately, the average reviewer has no way of knowing this. But that's hardly Google's fault, so they should be able to publish their papers. But it's not really reproducible. So this is a very gray area.
In order to publish a research paper in a journal, one of the requirement may be the reproducibility of results.
Your assumption is false. Only certain kinds of journals require reproducibility.
Classic examples of unreproducible results include:
Only the errors should not be published.
To add what others have already written, reproducibility in academic publishing mainly deals with the extent of how you have described your research methodology.
UPDATE: Un-reproducible results are largely frowned upon and a result of not describing the study properly. There is a place in science for experience reports and conjectures, however these should be framed as such.
Say, you are dealing with something unique like a supernova exploding. There is no way to recreate it in any practical sense. However, you can describe the star (type, size, distance, composition, environment, details of the explosion, etc etc) to provide the reader with enough details to understand what you are studying.
Similarly, you describe what kind of tools (telescopes, detectors etc.) you have used to collect data, what kind of methods you used to analyze the data, and so on. Thus, the reader can trace back your conclusions to data and to the circumstances of how the data was collected, and to the phenomenon being investigated.
Thus, the whole study may not be reproducible but parts of it can. Like, using the same equipment in the same way, applying the same data analysis methods, apply your reasoning to draw conclusions.
