How can we perpetuate reproducibility?

Question

Two years ago, Roger Peng discussed in his blog the ramifications of requiring reproducibility for the first producers of research objects, but not for reusers. He then proposed

What I think a good reproducibility policy should have is a type of "viral" clause. For example, the GNU General Public License (GPL) is an open source software license that requires, among other things, that anyone who writes their own software, but links to or integrates software covered under the GPL, must publish their software under the GPL too. This "viral" requirement ensures that people cannot make use of the efforts of the open source community without also giving back to that community.

I agree that having such a perpetuating policy element is very desirable, but would try to establish community norms around that, i.e. to get community organs like funders, publishers, institutions or learned societies to express support for the idea that reproducibility should trigger reproducibility. How best to phrase the core elements of such statements?

This question was originally posted at the Open Science SE private beta, due to be closed tomorrow.

Answer 1

A few ways to perpetuate reproducibility:

1. As a stipulation to publication, publishers could require the software and data used for a research paper to be made publicly available so that others could reproduce the results.
2. Academic institutions could provide repositories to allow its members to publish their code or data, to encourage making the code or data publicly available.

Difficulties regarding reproducibility requirements:

A major downside--perhaps a dealbreaker--is that there are significant issues with intellectual property. Large numerical models often take years to develop, benchmark and test, but once they are completed, they can often be used several times for several publications with minimal modification. Sharing the code they spent years developing in many cases would allow others to use the code to make discoveries before the original developers had a chance to do so. Allowing the original developers to protect their code gives them incentive to develop it.

Many data sets (especially in the social sciences) are protected/restricted because they contain classified, confidential, or personally-identifiable information. Moreover, many data sets that contain no personally identifiable information can actually be de-anonymized though careful computer analysis, so even protected data sets that have been "anonymized" are not necessarily safe to be released to the public.

Another difficulty is that some codes/scripts/routines may use commercial software (e.g. MATLAB, SPSS, etc.), so while the code themselves may be freely available, they might not necessarily be usable without paid software. This would exclude certain people from being able to reproduce results.

Some data sets are so large as to be an unreasonable burden on repositories. For example, I collaborated on a simulation that produced over 50 TB of data -- and there are other data sets that make 50 TB look tiny! It could cost a lot to maintain repositories on the order of hundreds of TB or several PB.

Summary

There are a few things that could be done to strongly encourage reproducibility, but they come at a cost. Releasing code, exact methodology, data sets, and other intellectual property would impact different fields in different ways and may not always be possible from a legal perspective. My guess is that data reproducibility will need to be treated on a field-by-field basis.

Answer 2

It's an economic problem, not a legal problem. Make the experiment cheap to do, and it will be widely copied. Provide funding or career advancement for reproduction, and people will do it. An unfunded legal mandate will just cause scientists to go elsewhere.