Why isn't it the norm to have research repeated immediately by other academics?

Question

The potential benefit to society of good research, and the potential costs of acting on bad research, seem to be extremely high in both directions.

Why is it the norm to simply peer review research, including the methodology, but not the norm to have the actual data collection and experiments re-run by other academics for cases where it is relatively low cost and fast to do so? E.g., where a second experiment/data collection run would take < 20% of the cost of the original research and < 3 months to conduct.

If this were standard practice on inexpensive, short term studies it could improve trust and integrity in research, so why isn't it standard practice?

Answer 1

Researchers make or break their reputation on ground-breaking, innovative, and high-impact work. Reproducing the results of another group provides essentially no benefit to any individual group (even if a culture of replication could arguably benefit the field).

More often than not, in my experience (in physics), reproduction of results is done as a first step in a new research project. The point of this is usually to check one's code/understanding before building up to something new. But it is not unusual for errors to be found and corrected this way. If a result is important enough to serve as a stepping stone for another research work, usually the key aspects of it are reproduced and it's not common for a visible, erroneous result to survive very long.

Answer 2

experiments re-run by other academics for cases where it is relatively low cost and fast to do so

The proportion of novel experiments which can be quickly and cheaply replicated is extremely small because the quick and cheap experiments were all done decades ago.

There are some exceptions in synthetic chemistry where, once the right reaction conditions are found, replication is quick and cheap.

When experiments can be replicated quickly and cheaply, and the experiments are actually valuable, then the experiments are, in fact, repeated promptly by many labs. Examples include CRISPR-Cas9, exfoliation of graphene, and perovskite solar cells. While this is not a "standard" practice, it does happen automatically; sometimes the replication is actually excessive.

Answer 3

Because not all research is interesting or important enough to other people for them to spend their own time and resources on it.

Because replication is frequently made difficult when original researchers won't share their methods or data with researchers who might try to disprove or discredit their work. ("We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.") Some journals have started imposing data-sharing requirements on contributors, but not all. So replication is often more difficult than original research where you have unfettered access to all the supporting materials.

For the same reason, some journals don't like replication studies that discredit research published in their pages. (There is a humorous essay here that describes an example.) They don't enhance the journal's reputation to the same degree as exciting original research, and are likely to be less interesting to their customers. That makes them harder to publish.

Because it may be seen socially as 'not nice' to trash somebody else's work. Many narrow research topics are 'small world' communities. You might find yourself applying to some professor for a job after previously discrediting his life's work. Or others may see a way to get preferential treatment in employment by falsely supporting a professor's work. There is scope for corruption.

Because conventional measures of academic research performance (number of papers published, journal impact, number of citations) don't measure it. If published papers didn't count towards your academic record until they had been independently replicated, research would look very different!

Because there is no need to check for replication before citing a paper's results in another work. Journals will allow you to depend on cited results that have not been independently verified.

Because replications are less likely to be cited, (especially if it results in the original paper being discredited) and so garner less academic credit.

Because there is no comprehensive systematic record of when and where papers and results have been replicated or discredited associated with the paper. (Counting reverse citations - where the original paper is considered to 'cite' the replication studies supporting or undermining it - would help.)

Because a culture of 'Argument from Authority' has built up that regards peer-reviewed journal papers as a scientific 'gold standard', rather than a work-in-progress in need of verification. They assume peer-reviewers have done all the detailed and comprehensive checking needed, rather than (in many cases) an unpaid expert spending a couple of hours briefly glancing through it to filter out the crazy. This means people often don't see the need to check for replication.

In summary - if people aren't motivated to replicate results by requiring it, they won't. Replication isn't required for someone to have a good publication record.

Answer 4

E.g., where a second experiment/data collection run would take < 20% of the cost of the original research and < 3 months to conduct.

Just because it would be cheaper and quicker than the original research, does not mean that the money and time are available to some other group.

Other groups have their limits on money and time that they are likely dedicating to finishing their own projects. It is very likely they are trying to find additional funding and to squeeze more efficient use of their time into their own projects, not looking for ways to burn their time and money checking someone else's work.

Answer 5

I'm going against the grain here and will say that any important result does get replicated, and this happens all the time.

Science and engineering are fields where we are always standing on the shoulders of giants. New science always builds upon older science. If the old science doesn't work, the new science doesn't either, so all important discoveries are necessarily verified since they can only be useful to other researchers when they are actually true, working, and correct - at least to some degree.

A large body of scientific work, of course, does not get replicated. This tells us only one thing - that said research had little intrinsic value to begin with. Nobody bothers to repeat something unless it helps them further some other goal. If the work is not worth repeating because it doesn't further any future scientific goals, then it is simply a dead-end curiosity, and it doesn't really matter if it was correct or not. It's not until new work begins to depend on it being correct that people start to care about whether or not is really is - and this is when science finds its faults and corrects those old crufty theories.

I suppose we can draw from this the conclusion that a lot of work is not repeated because much (most?) work in academia is either fruitless or useless and, while curious or interesting, doesn't actually contribute anything that helps the world get on with doing whatever it is we need or want to do. So nobody really gets too worked up about whether or not it was correct, because in the end many times it really doesn't matter - because nobody depends on it being correct, and nobody cares if it isn't. Some work is just not that interesting.

Answer 6

The difficulty with this lies in uniform application. The proposition makes a distinction between some types of research that should be verified and others that shouldn't (on time/cost basis for example).

This incentivises some research at the cost of others; it would be more fruitful/lucrative to do the kind of experiment that can't be easily repeated, because then the acceptance is not subjected to verification. That provision can be gamed by adding some component to the study that is difficult to replicate. In that way, essentially all experimemts could be facetiously shown to be difficult to exactly repeat.

The alternative is of course that all research be verified by reproduction; this would become unfair to those experiments that require very specialized equipment that is likely not available elsewhere.

Answer 7

Academia doesn't pay well enough for comparatively high qualified people to do the boring work of reproduction by default.

If you want such people to do boring work on a regular basis, you have to pay them much more.

That being said, scientists will often try to reproduce results they critically need, so you could say that, on the long run, natural scientific selection will actually end up probing important results for correctness over time anyway and there is no direct need to enforce that in the standard rules of operation.

Answer 8

Papers and research in general are a proof-of-concept of a certain idea. The results must be reproducible, but not necessarily reproduced: only time will tell if the proof of concept really works.

The peer-review should guarantee that the method is working as it claims to work, as well investigating if the claims are correct and the state-of-art is respected.

Research being reproducible should be a (natural) consequence of research itself, it should not be the goal of research.

Answer 9

The problem here, as in everything regarding the reproducibility crisis, lies in the incentives given to academics. As long as academics have (in this case, financial) incentives to focus on their career advancement, rather than on doing good and honest work, we won't get out of it.