18

A few days ago there was an article on the LSE blog about scientific reproducibility which made little sense to me until I realised they were equating 'low reproducibility' with 'scientific fraud':

http://blogs.lse.ac.uk/impactofsocialsciences/2016/07/21/could-blockchain-provide-the-technical-fix-to-solve-sciences-reproducibility-crisis/

I've tended to assume that most reproducibility issues are poor reporting of experiments, poor recording of external factors, poor statistical analysis inflating p-values etc but I realise that I don't have much evidence to back that up apart from personal experience. Are there any studies that report the relative proportion of irreproducible experimental results resulting from fraud versus poor experimental/statistical/reporting standards?

  • 7
    Could be very difficult to distinguish. For example, with statistical analysis, poor understanding, lazy implementation or genuine intent to deceive could all lead to exactly the same result. – user2390246 Mar 28 '17 at 9:55
  • 2
    Philosophically, the kind of study you have in mind seems nearly impossible to execute properly, since fraud is inherently hard to distinguish from mere lack of reproducibility. On top of poor experimenting/reporting, one aspect is that an irreproducible result can be always be an outcome of chance/randomness (p=0.05 means that it's randomness in 1 case out of 20). – lighthouse keeper Mar 28 '17 at 9:55
  • 5
    "Are there any studies that report the relative proportion of irreproducible experimental results resulting from fraud versus poor experimental/statistical/reporting standards?" How would you establish the ground truth for such a study? The only people that know whether a certain result was fraud or just sloppy are the authors, and they won't tell. – xLeitix Mar 28 '17 at 9:59
  • Sometimes, mistakes can be told from deliberate fraud, as the Poincaré and the fraudulent baker anecdote illustrates. – reinierpost Mar 28 '17 at 21:33
  • 1
    To go further than @lighthousekeeper's comment, you actually expect many unreproducible results to be published without any fraud---see publication bias. – Kimball Apr 22 '17 at 2:10
3

I very much doubt we "know' this in the empirical sense, as:

  1. As a scientific problem, the "reproducibility crisis" is fairly new, and we're still developing the methods to really understand it.
  2. It's very hard to distinguish fraud from error except in the most egregious cases. You can say, for example, that there's clearly some bias in a body of work in a field, but it's hard to say that any particular paper is clearly biased.

For example, is a convergence error that you chose to ignore because it was "pretty close" to the convergence criteria you set for a statistical model an error, or fraud? How about using Bayesian priors that turn out to be stronger than was probably justified? How do you distinguish fraud in the form of "nudging" a result over the line into significance from a largely insignificant effect being estimated with error that sometimes crosses the null.

Frankly, there's not even, IMO, a solid definition of "irreproducible". Is that "Finds a statistically significant result"? "Effect estimates are on the same side of the null"? "Our clinical/policy conclusions would be the same"?

Lacking even that, I'd assert it would be particularly difficult to attempt to assess fraud vs. error in any systematic fashion.

It would also be an exceedingly difficult study to run, as you'd need the cooperation of a bunch of people who comitted fraud.

-3

If you are asking what percent of experiments are reported as reproducible, when in fact they are not, I would think it is very, very low. The scientific method defines reproducible as an experiment that can be recreated, or performed by others, using the exact method. If these scientists have any conflicts of interests, they must be reported. After any trials of an experiment/study it is reviewed by another group of scientist. If a experiment isn't peer reviewed, it isn't a legitimate study to begin with. It is unlikely that you would be able to find a few, let alone a large portion of the scientific community to be in coercion and conspire to fake a theory.

  • That's clearly not what OP is asking, for two reasons: firstly, because the question talks about irreproducible results rather than irreproducible experiments, and secondly because the contrast is not reproducible vs irreproducible but "fraud versus poor experimental/statistical/reporting standards". Also, FWIW, in fields where the standard is a p-value of 0.05 and negative results aren't published (which is a whole separate issue) we would expect 5% of published results to be irreproducible in the best case, assuming no statistical errors. – Peter Taylor Oct 24 '17 at 16:00
-4

I consider advanced science as it's done today (a few separate teams tackling a particular issue with often very expensive equipment and/or materials) having great prospect for fraud by its very Nature. Consider, for example, science at the times of Copernicus and Galileo and science today. One can tell you it wasn't much of a cost to build a primitive telescope or laboratory then and anyone with more means could be scientist with a relatively low cost. Can you say the same for science today? How many people can spend the cash to build and equip (the costly part here) modern laboratory in anything compared to the same costs a century or two ago (or even a few decades ago)? (Exclude the millionaires-also consider some legal issues coming with modern and advanced laboratories.)

The more expensive and more profiled research gets the less people are actually available there to tell you the intentional fraud from the real sloppiness. As the research gets ever more profiled and more expensive to reproduce even the people who can argue the validness of a certain result are getting fewer and fewer, so how can anybody judge was it intentional or not? It's next to impossible. And it will probably get worse with time.

So, the study you suggest amounts to more than impossible. High research costs, lack of enough specialized personnel, greater abundance of claims to be falsified and not the least-doubtful interest in verifying them all amount to its impossibility with the advance of research efforts and costs. The more science is moving into an ever more profiled and costly activity, the more the very idea of falsifying every single claim someone makes into a paper is becoming implausible. And if you can't even spare the resources to clarify all mistakes how can you determine are they intentional or not? There may be some cases here and there where enough resources may be spent to clarify certain issues but you can't make a serious study out of those, right? Consider the amount of cases you are actually studying to the amount of possible cases of fraud. How can you get any statistically meaningful results if you are just picking up cases where you can determine the fraud was a real issue versus cases there may have been fraud? What standards will you use to discern this group from the group where you put errors made by sloppy research methods? What about a control of "impeccable research" (Is there even such a thing in any modern science?)? How can anybody devise "firm" criteria for putting anything in any of these categories when the effort to discern a false claim versus the number of possible claims is overwhelming. Just like nobody can pursue the validity of any claim due to the high costs and the lack of experienced enough personnel, so one can't make any "statistic" (except highly cease-sensitive and "narrow" field one) of the number of claims versus the number of fraud cases which could has any chances of being reliable?

P.S.As far as the issue of reproducibility is concerned I like to give one particular example. It's a bit hilarious but nevertheless I believe is on the right spot here :)

Consider the possibility there was some "large scale scientific conspiracy (e.g. a case of fraud)" concerning the results from the Large Hadron Collider at CERN. Now, let's say that every single participant in the experiments done there was part of some "grand scientific kabal" designed to mislead humanity about the very nature of our Universe. (I met people on the Internet who actually believe in this claim. :) Then, how can one be certain there is no fraud in the claims made by the scientists participating in the LHC experiments if there isn't any other such devices in the whole world? I tried to actually argue with "scientific kabal believers" such thing is impossible because there are simply too many people from too many countries doing research there and too many are watching the facility to serve as some "fraud facility" to lie to the poor people, but then I go back the argument that until one can build an LHC identical to the one at CERN in their backyard no one could be certain its results aren't fraud! Hilarious, isn't it? But, then I was dismayed how can one actually argue with someone putting up arguments like that? The funny thing is I couldn't. How can you convince a skeptic like that? And if you can't convince someone for such a big and visible experiment like the LHC how can you convince him other much less prominent experiments aren't fraud, too? But, then, if you can see fraud everywhere how can you discern it from mere sloppiness? What if you have doubts at the status of the very experts that have to do the discerning in itself? Then, what are you options?

The way I see it one can doubt endlessly in anything and everyone, then how can such an extreme skeptic ever achieve firm standards for anything which s/he can't get his/her hands on? And with the state current science is in this is practically impossible. Then, one should never be able to tell when there is a legitimate case of fraud and when things just weren't thought out well enough.

  • If you so much hate my answer and you're voting it down why don't YOU people write down one yourselves? At least, I made the effort to write an answer to the question-good or bad as it is! – Yordan Yordanov Mar 28 '17 at 22:58
  • 7
    Something is not always better than nothing. While we appreciate the effort, we don't necessarily appreciate the result. Downvotes are strictly an assessment of quality of the answer. They are not an assessment of the person posting the answer, nor of that person's effort. – R.M. Mar 29 '17 at 17:37
  • 1
    Most particularly, your answer is lacking in any references or hard numbers that the question is calling for specifically ("Are there any studies ..."). – R.M. Mar 29 '17 at 17:39
  • I think it answers the point why such studies are practically impossible to do-it's impossible to get good statistics of the numbers of actual fraud-related cases versus the number of possible cases, therefore the question the OP is asking can't be answered in the context of the modern state of organization of scientific research. This is the message I want to convey. If I don't-please, tell me why? May be I can learn something from this experience. I would also like to see an alternative answer to the question so I can compare it with mine, but in its absence what can I really tell? – Yordan Yordanov Apr 2 '17 at 12:34
  • 3
    therefore the question the OP is asking can't be answered — This is a good reason to downvote the question. – JeffE Apr 21 '17 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.