32

A follow up to this question.

There is an issue in science with experiments not being repeated. I asked if PhDs can be done on repeating experiments and the answer is kind of mixed. Most PhDs aren't that. I don't suspect that professors repeat experiments often.

So, who is supposed to repeat experiments in academia?

6
  • 2
    Journal of Irreproducible Results
    – Orion
    Commented Aug 5, 2016 at 19:23
  • 7
    "I asked if PhDs can be done on repeating experiments and the answer is kind of mixed" - Well, no, that's actually not quite what you asked. You asked if a PhD can be done solely on replicating the results of others. That's a subtly different question. Even if solely replicating the results of others wasn't enough for a PhD, that doesn't mean that PhD students won't/don't/shouldn't try to replicate the experiments of others. This question seems to be based on a faulty premise.
    – D.W.
    Commented Aug 5, 2016 at 23:48
  • 1
    You are. And you, who is reading this comment, too. Ideally, reviewers would be able to replicate experiments, but the usually have neither the data, nor the time, to do so. Commented Aug 7, 2016 at 7:37
  • 1
    I replicated the results of a paper for my undergraduate final project. We all got different papers and had to reproduce the results.
    – user21268
    Commented Aug 7, 2016 at 10:28
  • 2
    This is a great question! I usually have to rely at least on a single group being able to reproduce their results in several papers, then I would try to use the same method. It usually works. But in my topic it's hard to even reproduce my own results with sufficient accuracy. I would strongly ecourage this, in fact it would make sense for student profects to be based mostly on reproducing and checking the results of others (preferably new works). That way the students have the degree of certainty that their experiment should work, and the researchers can focus on obtaining new results mostly
    – Yuriy S
    Commented Aug 7, 2016 at 10:42

8 Answers 8

24

The gold standard is blind, independent replication. However, this almost never happens because it costs a similar amount (of materials and time) and there is less benefit to the researcher who repeats the experiments, so there is little incentive. Next down would be independent replication by another group, then replication within the same group. The bare minimum is that the experiment be described in sufficient detail to be reproduced. For example, the 'Instructions to Authors' in the Journal of Infection and Immunity (which is fairly typical) states:

the Materials and Methods section should include sufficient technical information to allow the experiments to be repeated.

Reproducibility is one of the most important issues when describing or reviewing a scientific paper (ref). Knowing that a result is less likely to be attributable to a chance combination of uncontrolled parameters, or a mistake on behalf of the scientist conducting the experiment or recording the experimental conditions, increases confidence that the result seen is a 'genuine' phenomenon. As Karl Popper, the founder of the scientific method, put it:

Non-reproducible single occurrences are of no significance to science.

(Popper 1959, The logic of scientific discovery. Hutchinson, London, United Kingdom.)

Unfortunately, most journals place a high premium on novelty and therefore it is more difficult to publish reproductions of previous studies, and when these are published they are often in less important journals. Funders, similarly, direct reviewers to score grant applications based, amongst other things, on novelty. This greatly reduces the incentive to replicate published studies - if you can't get funding to do something and you wouldn't be able to publish it anyway, it won't get done. Two other major and common obstacles to independent validation are:

  1. that it requires the involvement of another lab with the same resources and skills, which might be very rare;
  2. For experiments involving animals specifically, there is an ethical requirement to minimise the use of experimental animals.

The lack of reproducibility of experiments is a problem in most areas of science and awareness of this is growing. In 2012 this prompted the Reproducibility Initiative, where researchers can pay a fee for blind, independent replication prior to publication.

enter image description here

This sounded like a step in the right direction at the time but as far as I know awareness of the initiative remains fairly low, uptake has not been great and 'proper' reproduction remains very rare indeed.

2
  • 22
    "has less benefit" - this is the crux here. It does not have less benefit for the body of scientific knowledge as a whole. But the system is set up in a way in which it has less benefit for the scientist doing it. There is a known human bias towards novelty, and instead of trying to work around it (the way statistics are required to work around base rate fallacy and others), it works with it, e.g. by reviewers being required to grade submissions for novelty.
    – rumtscho
    Commented Aug 5, 2016 at 10:23
  • 2
    @rumtscho exactly. I will probably tinker with the text in the answer to make it clearer, but I absolutely mean the personal benefit to the scientist rather than the benefit to science as a whole. And publishing failed replication attempts is arguably more important, but even more rare; preregistration of experiments could go a long way to fixing this. It's required now for certain types of medical trials and Otteline Leyser proposed rolling it out to other areas during a debate on Radio 4 a few months ago, but it would be very unpopular and would have to be pushed by funders or publishers.
    – arboviral
    Commented Aug 5, 2016 at 11:19
15

I'm not sure anyone is supposed to repeat experiments, in any formalized way. In engineering, and I suppose physics, you end up repeating experiments because you are looking to use a method to solve a problem. So if someone has published a method to solve a particular problem, researchers will often try it in order to see how well it works for their system. Some methods work better than others, and I guess the best performing methods will be cited and used and become well known. Methods that don't work, or method with flaws and errors will tend to be forgotten over time.

2
  • 1
    That's how experiments (methods) are repeated in chemistry as well.
    – cbeleites
    Commented Aug 5, 2016 at 20:41
  • I often run into a problem at work, think "I'm sure someone has written a paper on this" and head to google scholar to see what the current research suggests.
    – user21268
    Commented Aug 7, 2016 at 10:41
12

Those whose research benefits from it

While it would be nice to repeat experiments, in practice we as a society don't really want1 scientists to repeat them just because, we want them to create new knowledge instead.

So experiments usually are repeated in three cases:

  1. When a researcher doubts the original experiment - if they distrust the outcome or the procedure, then replicating the experiment (possibly with modifications!) is useful science, since it creates new knowledge in addition to the initial result, no matter if the replication yields similar or different results. A negative result would definitely be publishable, and a positive result would be publishable iff the wider community also consider the first experiment as shaky and needing confirmation.
  2. When a researcher proposes an improved method or theory, they will often reproduce the previous experiment to make a proper comparison instead of just comparing reported numbers.
  3. Some experiments are replicated during teaching process, either to illustrate a concept to students, or to teach grad students state-of-art techniques before setting them on new experiments.

However, for experiments that are both hard to perform and seem trustworthy, there is no incentive to repeat them, it would not be particularly useful for the required effort. If you're really sure about the results that you'll get, then you don't gain any information by doing it, and that's not research anymore.

1Ignoring empty words, this is clearly illustrated by actions of funding bodies and university leadership who are setting the direction where scientists are applied.

4

As far as computer science is concerned, there are some journals that require experiments to be perfectly repeatable. In such cases they even require randomizer seeds to be explicitly stated during the review stage. However, not all journals have this sort of constraint, it differs from one reviewer to another.

Coming back to PhDs, in the end, it is nearly always up to the scholar to repeat the results to verify and refine them as needed. If you are working in a research team, you could use the published results produced or repeated by your peers.

4

Researchers don't start their work from scratch. New methods and objects appear to improve and extend previous research. If I read an article applying method X to object A and intend to apply method X to object B, I will:

  • replicate the experiment with method X and object A in order to check that my understanding of method X is correct

  • perform a new experiment with method X and object B

  • likely send a friendly notice to the authors of the original paper to tell them I've replicated their experiment.

3

Many results in particle physics are reproduced.

The detector systems in the large collider experiments are enormously flexible devices, and each new generation can (and does) reproduce and refine the work of the previous generation of machines as part of it's commissions and general data analysis. For instance, some of the earliest interesting results from ATLAS and CMS at the LHC were improved measurements of top-quark properties (which had previously only been measured by D0 and CDF at the Tevatron).

Something similar goes on with neutrino research, and low energy facilities also do a lot of that. My advisor was a nuclear physicist by original training who moved up the energy scale as time went by, and he once joked that nuclear physicists "do what particle physicist did twenty years ago, but ten to one hundred times as precisely" (which is too broad a claim in general, but has enough truth in the non-perturbative regime to make it good snark).

So, in fundamental physics the answer is "the people who work at the next generation facility".

2

In my field (sub-engineering), doing an experiment can be very costly and time consuming (to be honest, I haven't really heard of anyone repeating a full experiment!). So, it is very common for a PhD student to test 1-3 full scale specimens. For each experiment we do, we often develop a very detailed (3D, nonlinear, etc.) finite element model (that is validated against our experimental data). This model is also used to conduct a parametric study. In many occasions, this model is repeated by other researchers to study other parameters/case scenarios. So, in a way (and indirectly), the experiment is repeated!

2

I worked for some years in discrete optimization and must say that most computational experiments are not repeated in any rigorous way. From the scientific papers, it is usually not possible to reconstruct the method in detail because source code is not published.

In a very weak sense you could say that methods are repeated because people read about published methods and reimplement similar methods for their own optimization problems. So methods that appear in literature very often seem to be good.

On the other hand, you cannot really trust any "numerical experiments" that are published because usually nobody (not even reviewers) checked the implementations for bugs and logical errors.

I personally think that the requirement to always publish something "new" leads to a lot "inventions" of algorithms (that hide that fact that they are reformulated old algorithms), but prevents people from thoroughly analyzing the work of others.

You must log in to answer this question.