44

The potential benefit to society of good research, and the potential costs of acting on bad research, seem to be extremely high in both directions.

Why is it the norm to simply peer review research, including the methodology, but not the norm to have the actual data collection and experiments re-run by other academics for cases where it is relatively low cost and fast to do so? E.g., where a second experiment/data collection run would take < 20% of the cost of the original research and < 3 months to conduct.

If this were standard practice on inexpensive, short term studies it could improve trust and integrity in research, so why isn't it standard practice?

2
  • 63
    There is too much presure in academia to publish for people to have time for reproduction. Even if repeating experiments would become publishable, it would probably be seen as second class compared to new research. – Nick S May 30 at 2:22
  • Comments are not for extended discussion; this conversation has been moved to chat. Please read this FAQ before posting another comment. We can only move comments to chat once. – Massimo Ortolano Jun 2 at 9:02
57

Researchers make or break their reputation on ground-breaking, innovative, and high-impact work. Reproducing the results of another group provides essentially no benefit to any individual group (even if a culture of replication could arguably benefit the field).

More often than not, in my experience (in physics), reproduction of results is done as a first step in a new research project. The point of this is usually to check one's code/understanding before building up to something new. But it is not unusual for errors to be found and corrected this way. If a result is important enough to serve as a stepping stone for another research work, usually the key aspects of it are reproduced and it's not common for a visible, erroneous result to survive very long.

11
  • 4
    +1 for the info. "Researchers make or break their reputation on ground-breaking, innovative, and high-impact work. Reproducing the results of another group provides essentially no benefit to any individual group" .. wouldn't this logic also rule out peer reviews? i.e. if academics are completely preoccupied with exclusively new research, wouldn't they consider peer reviews an unnecessary distraction too? – stevec May 30 at 2:28
  • 22
    @stevec Many secretly do! But more to the point: peer-reviewing a paper is something you can do in an afternoon with experience and attention, and can be recorded on a CV as service work. Reproducing the result of another group can easily take months of the time of a PhD student or postdoc and by itself does not help anyone involved with their career. – Andrew May 30 at 2:30
  • 16
    @stevec Institutional inertia. You would essentially need to convince funding agencies to fund work on replication studies, and you would need to convince high quality journals to devote space to publishing these studies, and you would need to convince hiring committees to look for this work, and you would need all of these bodies to accept that there would be less time and resources spent on work that would lead to new research. – Andrew May 30 at 2:34
  • 9
    Plus people get into science because, at least on some level, they think they will make a ground breaking discovery, not to spend months of their time trying to reconstruct someone else's work from a sketchily written methods section. – Andrew May 30 at 2:36
  • 45
    My personal theory on this is that you are running into a classic problem in economics. There may be global solutions where many parties cooperate that are optimal for the community, but they are "unstable" to individuals not cooperating to maximize their personal interest. The current system is a kind of local equilibrium responding to various sources of pressure where no one can do significantly better on their own. But maybe I am too cynical. – Andrew May 30 at 2:38
31

experiments re-run by other academics for cases where it is relatively low cost and fast to do so

The proportion of novel experiments which can be quickly and cheaply replicated is extremely small because the quick and cheap experiments were all done decades ago.

There are some exceptions in synthetic chemistry where, once the right reaction conditions are found, replication is quick and cheap.

When experiments can be replicated quickly and cheaply, and the experiments are actually valuable, then the experiments are, in fact, repeated promptly by many labs. Examples include CRISPR-Cas9, exfoliation of graphene, and perovskite solar cells. While this is not a "standard" practice, it does happen automatically; sometimes the replication is actually excessive.

1
  • 1
    Also, with examples like the ones you mention, there's the possibility of interesting (and potentially lucrative :-)) further research. For instance, Group A demonstrates that it's possible to make perovskite solar cells, then Groups B, C, & D all jump on the idea because of the possibility of making better ones. – jamesqf May 30 at 17:25
26

Because not all research is interesting or important enough to other people for them to spend their own time and resources on it.

Because replication is frequently made difficult when original researchers won't share their methods or data with researchers who might try to disprove or discredit their work. ("We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.") Some journals have started imposing data-sharing requirements on contributors, but not all. So replication is often more difficult than original research where you have unfettered access to all the supporting materials.

For the same reason, some journals don't like replication studies that discredit research published in their pages. (There is a humorous essay here that describes an example.) They don't enhance the journal's reputation to the same degree as exciting original research, and are likely to be less interesting to their customers. That makes them harder to publish.

Because it may be seen socially as 'not nice' to trash somebody else's work. Many narrow research topics are 'small world' communities. You might find yourself applying to some professor for a job after previously discrediting his life's work. Or others may see a way to get preferential treatment in employment by falsely supporting a professor's work. There is scope for corruption.

Because conventional measures of academic research performance (number of papers published, journal impact, number of citations) don't measure it. If published papers didn't count towards your academic record until they had been independently replicated, research would look very different!

Because there is no need to check for replication before citing a paper's results in another work. Journals will allow you to depend on cited results that have not been independently verified.

Because replications are less likely to be cited, (especially if it results in the original paper being discredited) and so garner less academic credit.

Because there is no comprehensive systematic record of when and where papers and results have been replicated or discredited associated with the paper. (Counting reverse citations - where the original paper is considered to 'cite' the replication studies supporting or undermining it - would help.)

Because a culture of 'Argument from Authority' has built up that regards peer-reviewed journal papers as a scientific 'gold standard', rather than a work-in-progress in need of verification. They assume peer-reviewers have done all the detailed and comprehensive checking needed, rather than (in many cases) an unpaid expert spending a couple of hours briefly glancing through it to filter out the crazy. This means people often don't see the need to check for replication.

In summary - if people aren't motivated to replicate results by requiring it, they won't. Replication isn't required for someone to have a good publication record.

4
  • 1
    Very nice. And, you aren't going to get a doctorate for this, almost assuredly. – Buffy May 30 at 15:07
  • Fascinating and brutal! Interested to know why the system hasn't self-corrected from this state? Other systems sometimes improve over time via self regulation or inquiries of various sorts. For example, various industries (e.g. telco, financial, fisheries etc) often go through multi-year/decade mires often followed by inquiries and subsequent improvements. Why don't the obviously imperfect incentive structures you outlined get addressed? Is the system too complicated? Are there vested interests in keeping the status quo? Or is imperfect system already the best it can be? – stevec May 30 at 23:48
  • 4
    @stevec Strict scientific principle is all very well, but scientists are human too, we all have to pay the bills, and get along with our colleagues. Industries have a strong and direct financial incentive to root out sloppy science that overrides this, (and even so are by no means perfect) - academics funded by government bureaucracies that are themselves staffed by more senior committee-minded scientists not so much. (It's even worse when science gets political.) Evolution needs natural selection. It will take an even more major scandal than we've yet seen to trigger those inquiries/reform. – Nullius in Verba May 31 at 1:02
  • 1
    @stevec You may be interested in the book "Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope, and Wastes Billions" by Richard Harris. – Anyon May 31 at 15:26
7

E.g., where a second experiment/data collection run would take < 20% of the cost of the original research and < 3 months to conduct.

Just because it would be cheaper and quicker than the original research, does not mean that the money and time are available to some other group.

Other groups have their limits on money and time that they are likely dedicating to finishing their own projects. It is very likely they are trying to find additional funding and to squeeze more efficient use of their time into their own projects, not looking for ways to burn their time and money checking someone else's work.

7

I'm going against the grain here and will say that any important result does get replicated, and this happens all the time.

Science and engineering are fields where we are always standing on the shoulders of giants. New science always builds upon older science. If the old science doesn't work, the new science doesn't either, so all important discoveries are necessarily verified since they can only be useful to other researchers when they are actually true, working, and correct - at least to some degree.

A large body of scientific work, of course, does not get replicated. This tells us only one thing - that said research had little intrinsic value to begin with. Nobody bothers to repeat something unless it helps them further some other goal. If the work is not worth repeating because it doesn't further any future scientific goals, then it is simply a dead-end curiosity, and it doesn't really matter if it was correct or not. It's not until new work begins to depend on it being correct that people start to care about whether or not is really is - and this is when science finds its faults and corrects those old crufty theories.

I suppose we can draw from this the conclusion that a lot of work is not repeated because much (most?) work in academia is either fruitless or useless and, while curious or interesting, doesn't actually contribute anything that helps the world get on with doing whatever it is we need or want to do. So nobody really gets too worked up about whether or not it was correct, because in the end many times it really doesn't matter - because nobody depends on it being correct, and nobody cares if it isn't. Some work is just not that interesting.

7
  • Nice expansion of the terse second argument of my response. +1. – Captain Emacs May 31 at 15:52
  • 2
    @CaptainEmacs Yes, a few answers (like yours, +1) touched on this, but I felt the presentation could be inverted - (ie: frame challenge - we DO repeat important results, with caveats that garbage tends to get ignored). The fallacy in academia is usually to fail to appreciate just how much research is not particularly useful, whether it's correct or not. I think that clouds the perception of how repeating experiments works. – J... May 31 at 16:03
  • Quite so. It reminds me a bit of Pauli's "Not even wrong..." criticism. On the other hand, it's often hard to see which research is actually fruitful in the long run, which one is rubbish and which one is simply dormant. – Captain Emacs May 31 at 16:17
  • 1
    Agreed - that's why I said "it reminds me" rather than "it's like" :-) On the other hand, I detest having to give significance statements as a reviewer, as they pretend to be objective whereas only the future can be. Even Adleman from RSA seems to have thought their paper was quite a minor result with no major relevance. – Captain Emacs May 31 at 16:26
  • 3
    This answer matches my experience. My group normally replicate at least one or two plots from a paper we are building on, just to check our own data/equations/algorithms are good. Once we actually couldn't replicate part of a plot and we did discover an oversight in the other paper's methodology. It was minor, but important enough to mention in our paper. – Clumsy cat May 31 at 18:52
4

The difficulty with this lies in uniform application. The proposition makes a distinction between some types of research that should be verified and others that shouldn't (on time/cost basis for example).

This incentivises some research at the cost of others; it would be more fruitful/lucrative to do the kind of experiment that can't be easily repeated, because then the acceptance is not subjected to verification. That provision can be gamed by adding some component to the study that is difficult to replicate. In that way, essentially all experimemts could be facetiously shown to be difficult to exactly repeat.

The alternative is of course that all research be verified by reproduction; this would become unfair to those experiments that require very specialized equipment that is likely not available elsewhere.

1
  • 1
    +1 I agree it could create some of its own perverse incentives (e.g. to avoid having research reproduced). But I'd also guess it could create an excellent avenue for confident researchers to demonstrate the robustness of their findings. – stevec May 30 at 4:06
3

Academia doesn't pay well enough for comparatively high qualified people to do the boring work of reproduction by default.

If you want such people to do boring work on a regular basis, you have to pay them much more.

That being said, scientists will often try to reproduce results they critically need, so you could say that, on the long run, natural scientific selection will actually end up probing important results for correctness over time anyway and there is no direct need to enforce that in the standard rules of operation.

1
  • 1
    Scite.ai attempts this in a meta-level, classifying publications citing prior work as either a) supporting, b) mentioning, c) contrasting. There are some publishers already using their service regularly or even are partners (see). But depending on your field, some big ones (by number of journals / by impact of their journals / an other criterion) are evidently missing. – Buttonwood May 30 at 16:27
0

Papers and research in general are a proof-of-concept of a certain idea. The results must be reproducible, but not necessarily reproduced: only time will tell if the proof of concept really works.

The peer-review should guarantee that the method is working as it claims to work, as well investigating if the claims are correct and the state-of-art is respected.

Research being reproducible should be a (natural) consequence of research itself, it should not be the goal of research.

-1

The problem here, as in everything regarding the reproducibility crisis, lies in the incentives given to academics. As long as academics have (in this case, financial) incentives to focus on their career advancement, rather than on doing good and honest work, we won't get out of it.

1
  • The response is somewhat polemic. I do not see anything not good or dishonest in not being interested reproducing some mediocre uninteresting result of some fellow researcher if one believes one can find interesting and relevant results oneself instead. I'd prefer scientists to rather try to be productive than to demonstrate that some pointless work is indeed pointless. The interesting stuff will ultimately be checked. – Captain Emacs May 31 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.