Do we know what proportion of scientific reproducibility issues are fraud-related?

Question

A few days ago there was an article on the LSE blog about scientific reproducibility which made little sense to me until I realised they were equating 'low reproducibility' with 'scientific fraud':

http://blogs.lse.ac.uk/impactofsocialsciences/2016/07/21/could-blockchain-provide-the-technical-fix-to-solve-sciences-reproducibility-crisis/

I've tended to assume that most reproducibility issues are poor reporting of experiments, poor recording of external factors, poor statistical analysis inflating p-values etc but I realise that I don't have much evidence to back that up apart from personal experience. Are there any studies that report the relative proportion of irreproducible experimental results resulting from fraud versus poor experimental/statistical/reporting standards?

Answer 1

I very much doubt we "know' this in the empirical sense, as:

1. As a scientific problem, the "reproducibility crisis" is fairly new, and we're still developing the methods to really understand it.
2. It's very hard to distinguish fraud from error except in the most egregious cases. You can say, for example, that there's clearly some bias in a body of work in a field, but it's hard to say that any particular paper is clearly biased.

For example, is a convergence error that you chose to ignore because it was "pretty close" to the convergence criteria you set for a statistical model an error, or fraud? How about using Bayesian priors that turn out to be stronger than was probably justified? How do you distinguish fraud in the form of "nudging" a result over the line into significance from a largely insignificant effect being estimated with error that sometimes crosses the null.

Frankly, there's not even, IMO, a solid definition of "irreproducible". Is that "Finds a statistically significant result"? "Effect estimates are on the same side of the null"? "Our clinical/policy conclusions would be the same"?

Lacking even that, I'd assert it would be particularly difficult to attempt to assess fraud vs. error in any systematic fashion.

It would also be an exceedingly difficult study to run, as you'd need the cooperation of a bunch of people who comitted fraud.

Answer 2

If you are asking what percent of experiments are reported as reproducible, when in fact they are not, I would think it is very, very low. The scientific method defines reproducible as an experiment that can be recreated, or performed by others, using the exact method. If these scientists have any conflicts of interests, they must be reported. After any trials of an experiment/study it is reviewed by another group of scientist. If a experiment isn't peer reviewed, it isn't a legitimate study to begin with. It is unlikely that you would be able to find a few, let alone a large portion of the scientific community to be in coercion and conspire to fake a theory.

Answer 3

I consider advanced science as it's done today (a few separate teams tackling a particular issue with often very expensive equipment and/or materials) having great prospect for fraud by its very Nature. Consider, for example, science at the times of Copernicus and Galileo and science today. One can tell you it wasn't much of a cost to build a primitive telescope or laboratory then and anyone with more means could be scientist with a relatively low cost. Can you say the same for science today? How many people can spend the cash to build and equip (the costly part here) modern laboratory in anything compared to the same costs a century or two ago (or even a few decades ago)? (Exclude the millionaires-also consider some legal issues coming with modern and advanced laboratories.)

The more expensive and more profiled research gets the less people are actually available there to tell you the intentional fraud from the real sloppiness. As the research gets ever more profiled and more expensive to reproduce even the people who can argue the validness of a certain result are getting fewer and fewer, so how can anybody judge was it intentional or not? It's next to impossible. And it will probably get worse with time.

So, the study you suggest amounts to more than impossible. High research costs, lack of enough specialized personnel, greater abundance of claims to be falsified and not the least-doubtful interest in verifying them all amount to its impossibility with the advance of research efforts and costs. The more science is moving into an ever more profiled and costly activity, the more the very idea of falsifying every single claim someone makes into a paper is becoming implausible. And if you can't even spare the resources to clarify all mistakes how can you determine are they intentional or not? There may be some cases here and there where enough resources may be spent to clarify certain issues but you can't make a serious study out of those, right? Consider the amount of cases you are actually studying to the amount of possible cases of fraud. How can you get any statistically meaningful results if you are just picking up cases where you can determine the fraud was a real issue versus cases there may have been fraud? What standards will you use to discern this group from the group where you put errors made by sloppy research methods? What about a control of "impeccable research" (Is there even such a thing in any modern science?)? How can anybody devise "firm" criteria for putting anything in any of these categories when the effort to discern a false claim versus the number of possible claims is overwhelming. Just like nobody can pursue the validity of any claim due to the high costs and the lack of experienced enough personnel, so one can't make any "statistic" (except highly cease-sensitive and "narrow" field one) of the number of claims versus the number of fraud cases which could has any chances of being reliable?

P.S.As far as the issue of reproducibility is concerned I like to give one particular example. It's a bit hilarious but nevertheless I believe is on the right spot here :)

Consider the possibility there was some "large scale scientific conspiracy (e.g. a case of fraud)" concerning the results from the Large Hadron Collider at CERN. Now, let's say that every single participant in the experiments done there was part of some "grand scientific kabal" designed to mislead humanity about the very nature of our Universe. (I met people on the Internet who actually believe in this claim. :) Then, how can one be certain there is no fraud in the claims made by the scientists participating in the LHC experiments if there isn't any other such devices in the whole world? I tried to actually argue with "scientific kabal believers" such thing is impossible because there are simply too many people from too many countries doing research there and too many are watching the facility to serve as some "fraud facility" to lie to the poor people, but then I go back the argument that until one can build an LHC identical to the one at CERN in their backyard no one could be certain its results aren't fraud! Hilarious, isn't it? But, then I was dismayed how can one actually argue with someone putting up arguments like that? The funny thing is I couldn't. How can you convince a skeptic like that? And if you can't convince someone for such a big and visible experiment like the LHC how can you convince him other much less prominent experiments aren't fraud, too? But, then, if you can see fraud everywhere how can you discern it from mere sloppiness? What if you have doubts at the status of the very experts that have to do the discerning in itself? Then, what are you options?

The way I see it one can doubt endlessly in anything and everyone, then how can such an extreme skeptic ever achieve firm standards for anything which s/he can't get his/her hands on? And with the state current science is in this is practically impossible. Then, one should never be able to tell when there is a legitimate case of fraud and when things just weren't thought out well enough.