28

Background: the answers and comments on this question prompted this question.

I know that when a new scientific theory is published there is a rush for people to devise experiment to prove or disprove it. However there is lot of research that involves original experiments that are not necessarily related to new not-yet-proven/accepted theories.

Now, obviously science relies heavily on peer-verification of theories and experiments, but are experiments today actually verified by independent peers?

The thread I linked to made me grow some doubts: if an experiment has so much trouble being funded, how is it possible to scientifically verify the results of already published data, especially of high-cost experiments. If there is no incentive in doing that verification, how can articles describing a new experiment be validated scientifically?

Disclaimer: sorry for the possible dumb question, but I had only a brief experience in academia at the beginning of my career (and it was in SW engineering, where "experiments" are way less common than proof of concepts software, so verification is a far less difficult subject there).

As a further explanation of my doubts, the last time I remember an experiment being rejected because it couldn't be validated was the thing about cold fusion by Fleischman and Pons, back in the late 80s. At least that was a case that reached mainstream media. I guess such events (both validation and rejection) are more common in the specialized journals, but I don't remember any other "big event" that reached the mainstream media since then.

Please, note well that I'm not claiming that experimental results never get published by the mainstream media, here I'm focusing on the validation by peers that are independent from the original authors/scientists.


Maybe I wasn't too clear, possibly because I didn't use the right terminology.

I'm not particularly interested in the review process details (although I welcome any correction about my lack of terminology), but in actual scientific validation of experimental results.

What I really care for is whether or not published scientific experimental results are effectively put under experimental scrutiny by the scientific community.

17
  • 3
  • 1
    I think the problem with your question is the way it is phrased it appears that you think the academic peer review process involves experimental scrutiny. This occurs because the title of your question with the emphasized "actually" implies that this used to be the case or is claimed to be the case.
    – Bryan Krause
    Commented May 17, 2023 at 14:06
  • 1
    @LorenzoDonatisupportUkraine Best to be most blunt and concrete when initially writing the question. :) We're in a bit of a pickle here since there are numerous existing answers, so it's kind of too late to change the question. Being very concrete, are you asking specifically about novel methodology? Because this is quite different from simply an original experiment.
    – Bryan Krause
    Commented May 17, 2023 at 14:39
  • 2
    @LorenzoDonatisupportUkraine I'm not sure you're noticing then the difference between peer review and the broader scientific discourse; the former is based a lot more on trust than the latter. But yes, you've correctly discovered that science is influenced by economics, politics, friendship, etc: science is done by people, and people are influenced by all those things.
    – Bryan Krause
    Commented May 17, 2023 at 14:49
  • 6
    I think you should do some research into the "Replication Crisis." Ongoing research by some Psychologists have found that they are unable to reproduce most published psychology findings, or they are but they find the effect is 1 to 2 magnitudes lower then the published effect... Some big experiments to mention are the Stanford prison experiment (Which we now know to be a poisoned experiment).
    – Questor
    Commented May 17, 2023 at 20:32

6 Answers 6

20

Others have discussed the official process.

Generically, the thing is, the process is messy. There is no megalithic authority presiding over science, nor academia in general. We don't have a "king of science." There is no "pope of academia."

As such, it is possible for a false result to remain unchallenged in the literature for some time. This can be due to honest mistakes or actual fraud or various combinations.

The process of science eventually turns to re-examine old ideas. People are hungry for publication. There are teams and groups and milling armies of people hungry for publication. They search through old journal articles, read them, sit and think "I could re-do that experiment in a couple weeks with stuff in storage at the lab." And many variations on this. They re-do the arithmetic in published articles. They request the raw data. They compare other experimental results reported at other labs.

So, at any given moment, it is quite likely the literature contains many errors, mistakes, and a few (hopefully a very small number) of actual frauds. And science will try to correct them.

As Dara O'Briain said: Science knows it does not know everything. Otherwise it would stop.

Another thing that tends to happen is that a result becomes irrelevant due to advances that move us past it. As a trivial example that nobody probably cares about: Suppose somebody claimed they could make electronic vacuum tubes to perform a certain way, but that their result was wrong. The advent of transistors moved us away from vacuum tubes, so nobody cares anymore about the performance of tubes. Possibly such an incorrect result from 60 years ago might hang around in the journals and never get challenged. So there is a category of knowledge that one should take extra caution about accepting as authoritative, because the process of science dropped the topic.

So you can recognize a healthy science endeavour by the presence of people questioning the results. They need to be doing it in a well based manner, not simply sticking their fingers in their ears and going "la la la." They need to understand the issues, the theories, the current state of things. And then they need to be free to ask questions and make critiques. If there is nobody questioning a result that is a warning that the science has gone moribund.

1
  • 8
    This is an important point. If the science "matters" (loaded term), you can bet it will be attacked and investigated thoroughly, like Pons and Fleischman. Invalidating those findings is important and publishable by itself. But if it doesn't "matter", like vacuum tubes, nobody's going to spend the time and money to bother. Science is a dynamic conversation where the interesting ideas keep going, and the boring tangents get left behind.
    – mightypile
    Commented May 17, 2023 at 22:02
27

There are two related but separate processes in play here.

On the one hand, papers are peer reviewed, i.e., when you submit a paper describing your experiment, results and conclusions to a conference or journal, two to three peers will review it. They review the paper - they typically don't have the time, funding or other requirements to actually re-do your experiment. They will check whether your arguments make sense, whether the experiment you describe can in principle be used to address your research question, maybe run a small simulation.

On the other hand, people will try to replicate your experiment, i.e., run a similar experiment and see whether they get the same results. (Typically they will run a modification, because straight-up replications are very hard to publish, and publications are important to academics.) If you have an exciting result, many people will try to replicate and extend it. If then most of these people cannot replicate your results (take a look at the "replication crisis in psychology"), the excitement of your results will wane.

16
  • 10
    In the end, it very much depends, especially on the field. Very few fields allow you to exactly replicate an experiment down to the last detail, so some changes between experiment runs are inevitable. What people look for in replications is mainly a consistency in the patterns, not necessarily a perfect match: do all the results point in the same direction, or did someone get a surprising positive result, but the next 20 similar experiments all got a negative result? Yes, there is some subjectivity involved in when experiment B is a "true" replication of experiment A. Commented May 17, 2023 at 11:33
  • 18
    It can actually be bad if everyone does the experiment in exactly the same way. Repeating an experiment with changes helps to confirm that a particular part of the experimental methodology isn't having unforeseen effects on the results. Commented May 17, 2023 at 13:00
  • 4
    @LorenzoDonatisupportUkraine in reality, people who read the paper just have to trust the experiment was actually done. If the writer of the paper lied, it creates a lot of drama and problems for them later... if they ever get caught. I'm sure many didn't get caught. Commented May 17, 2023 at 19:30
  • 8
    @LorenzoDonatisupportUkraine If you do find replication with a slightly different experimental setup, it's even better for the underlying theory. We usually try to generalize in science, we aren't usually interested in findings that can only be observed under one very specific set of experimental conditions. If you don't find replication with a different setup, it is on the experimenter to convincingly explain why the changes in setup shouldn't have affected the results. Commented May 17, 2023 at 19:46
  • 5
    @LorenzoDonatisupportUkraine In this case we have still obtained knowledge about the validity of theory T. That the wrong people might be credited with the 'discovery' is a shame, but only tangential to the purpose of the scientific process. That theory T was thought to be verifiable because of a mistake (experiment EA) is part of the messiness of science in practice. If experiments EB and EC would have disagreed with theory T, this would cast doubt on experiment EA, and it might then be (reproduced and) discarded as false. This is all part of the process.
    – Servaes
    Commented May 18, 2023 at 13:49
21

When laypeople are introduced to the idea of replication, they imagine scientists constantly redoing every paper that comes out, to "check it". At least I did. This is not the case in most mainstream sciences. The closest thing is when a paper is basically announcing the release of a software - then conceivably a lot of people might give it a try, since it's usually easy (or is it?) to just install and run a software on your computer.

This does not mean there is no replication. To be sure, there is a replication crisis, in the sense that we have too many studies that should be replicable but aren't. But it's important to remember that a lot of science can and is replicated.

The replication does not take the form of a researcher going to their PI, and submitting a 20-page document titled "Proposal for Comprehensive Replication of Jones et al. 2022". Replication is rarely a discrete project, it is embedded in novel research. The rare exception is extremely high profile and controversial work - for example, the Yamanaka induced pluripotency protocol inspired many outright reproduction attempts.

A lot of science these days is not "from scratch", but building upon and refining previous studies. Say Smith 2019 discovered that red M&Ms cure cancer in mice. Murphy 2024 might then test different chemical extracts of red M&Ms across various mutant mice, and discover that a specific chemical is responsible for the effect by acting on a certain gene - such a study might often begin with a "sanity check" by repeating a simple version of the original Smith 2019 experiment. Indeed the Smith 2019 regimen may serve as a baseline or control for evaluating the additional claims of Murphy 2024.

Another common situation is when published results are assumed to be true, and an experiment is set up such that if the assumption is false the experiment will produce nonsense results. For example, imagine that Dr. Murphy decides to feed some mice only the filling of red M&Ms, while other mice are fed only the shell. It's expected that either group will either be cured of cancer or not. But if all mice die of chemical poisoning, that obviously raises some questions about the results claimed in Smith 2019. It is often possible, with clever experimental design, to "cover" validation of past results as well as probe the actual hypothesis of interest.

You also mention cost. However, you make the false assumption that reproduction always costs as much as the original experiment. Think of it this way - if you decided to sail from Europe to America, would you need to be an equal to Columbus? A big sink of money, and time, therefore money, in original research is trial and error. There are a lot of unknowns about what experiment would work, so it costs a lot to perform it. However, reproducers can skip all that - with the benefit of hindsight they can avoid the false starts.

2
  • 3
    Thank you for the extensive answer. Particularly insightful the final part about skipping false starts. Commented May 18, 2023 at 8:08
  • ... it's usually easy (or is it?) to just install and run a software on your computer - the nicest situation from my experience is when researchers provide a Jupyter notebook with their analysis, which anyone can run without needing to install custom software, preferably hosted on Colab; I wish this was more common.
    – yoniLavi
    Commented May 20, 2023 at 12:18
11

but are experiments today actually verified by independent peers?

In some (rare) cases, yes.


The examples that I have are in Computer Science, where there is a big push lately for articles to share their "artifacts", which are environments that allow to reproduce their results.

Major associations such as the IEEE and the ACM now have their own "badges" (IEEE badge and ACM badges) that are awarded only if the artifact is publicly available, if the results can be reproduced, and if the environment is "re-usable" (read, it's easy to tweak their plat-form to test another hypothesis).

Major conferences such as Discotec or POPL now have their "artifact evaluation committee" whose main goal is to check that the results from the papers can be reproduced.

5
  • By "artifact" do you mean something that also include the source code? Commented May 17, 2023 at 13:39
  • 2
    @LorenzoDonatisupportUkraine Absolutely. An artifact generally contain the source code, instructions to execute the program(s) and various configuration files. One example (disclaimer: that I contributed to) can be found at github.com/statycc/loop-fission#reproducing-results .
    – Clément
    Commented May 17, 2023 at 17:56
  • 3
    I can confirm this is the case, I did a PhD in computer science (machine learning) and towards the end of my candidature it became increasingly difficult to publish without also including a link to source codes, datasets and other artifacts that might be needed to verify the results. In general, this is a good thing, for me it was a challenge as my PhD was industry-sponsored making it hard to comply. Commented May 17, 2023 at 22:44
  • 1
    @Clément Thanks for the added detail. Commented May 18, 2023 at 8:09
  • 1
    @DavidWaterworth Interesting POV. Thanks. Commented May 18, 2023 at 8:10
8

No, peer-reviewers usually just read the manuscript. However with a lot of experience reviewers can often tell from the presented results and the methods description if the research is valid.

If researchers have doubts about the published results they usually do a replication study and then write a letter to the editor.

8

I might be worth thinking about science from a Kuhn-ian point of view, rather than a Popperian one.

Start with the somewhat surprising assumption that all scientific conclusions are really models of how the world works, and that all models will eventually be shown to be false to some degree.

In Kuhn's model scientists conduct studies under the assumptions of the current Paradigm (i.e. the set of models that people assume to be true). As long as the results of those studies produce data that makes sense under the current paradigm, they support its truth/usefulness of the models that make up the current paradigm. To the extent that those studies propose new models, they become part of the paradigm. Thus we build up "sandcastles", each new theory based on the validty of the those that came before it, and the success of the new thoeries support the validity of the old. But over time, anomolous results accumulate that can't be explained by the current paradigm. At first these can just be ignored, or explained away, but eventually the weight of anomolies becomes so heavy that the models on which the whole paradigm collapses and you have a scientific revolution, or paradigm-shift. The things that survive this shift will continue to form the foundations on which the new sandcastle is built, and overtime will be compressed to form sandstone.

As well as happening at a macro-level (relativity replacing newtonian physics for eg), it also happens at a micro-level. If a paper is published proposing X and providing an experiment that supports it, rather than trying to replicate those results, someone will say "well, if X is true, then perhaps Y is also true", and they will conduct an experiment assuming X is true. If those results are positive, they lend support to X, if they fail, then the probably don't in one go disprove X, but they might start to add question marks to it. If there are enough such results, people will start looking round for other theories to based their investigations on. This usually doesn't happen in a big bang (like it would do at the macro-level), but mostly just people drifting away from believe X - its not demosntrated false, it just that those that don't base their science on it tend to be more successful.

One downside of this process is that it is slow, and at any one time only those with a deep knowledge of the whole field will really have a good idea of what the field believes to be true or not at any given time. In fact, at any given time it would be very hard for someone to say "here is a list of things that people in field A believe to be definately true". We will only really know what was a good call and what not decades or even centuries into the future.

This is of course no good when your results are needed for practical purposes (like treating diseaes). A couple of things to observe here is that even if we can't say for certain that X is true, if you had to bet your life on X or not X, you'd be better off betting on X, even if you only have a qualified belief in it being true. The second is that the medical and health fields are fields were replications really do happen. There are often multiple clinical trials into the effect of a particular exposure or class of drugs. These are generally collected together in meta-analyses which collate all to evidence for a single question to see what the evidence is in aggregate.

9
  • 5
    For better or worse, something often misunderstood by non-scientists is just how much easier it is to do science showing whether a treatment works than to figure out how it works. Another reason that medicine and basic science tend to use different approaches to replication.
    – Bryan Krause
    Commented May 18, 2023 at 14:56
  • Thanks for the interesting POV. I didn't know anything about the Kuhnian POV. Commented May 18, 2023 at 15:37
  • @BryanKrause That's definitely true and it's largely due IMO because you have (a) strict regulations for researching and marketing new treatments or medical devices and (b) the prospect of huge lawsuits if something goes horribly wrong due to negligence, incompetence or sheer fraud. Commented May 18, 2023 at 15:43
  • 4
    @LorenzoDonatisupportUkraine No, it's because it's a simple experiment to say does X change Y. You give X, you measure Y. Mechanism is far more difficult to assess, and many tests of mechanism are inconclusive, you might be measuring a separate correlate rather than something in the causal chain, and so most subsequent experiments will not try to exactly test the same thing again, there are just too many potential facets and targets.
    – Bryan Krause
    Commented May 18, 2023 at 16:36
  • 1
    @LorenzoDonatisupportUkraine Every child knows that if they throw a ball up it falls down again, but figuring out How Gravity Works is complicated. But you can still do a lot of useful stuff only with the partial result that stuff that goes up comes back down. Commented May 19, 2023 at 14:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .