3

Two years ago, Roger Peng discussed in his blog the ramifications of requiring reproducibility for the first producers of research objects, but not for reusers. He then proposed

What I think a good reproducibility policy should have is a type of "viral" clause. For example, the GNU General Public License (GPL) is an open source software license that requires, among other things, that anyone who writes their own software, but links to or integrates software covered under the GPL, must publish their software under the GPL too. This "viral" requirement ensures that people cannot make use of the efforts of the open source community without also giving back to that community.

I agree that having such a perpetuating policy element is very desirable, but would try to establish community norms around that, i.e. to get community organs like funders, publishers, institutions or learned societies to express support for the idea that reproducibility should trigger reproducibility. How best to phrase the core elements of such statements?

This question was originally posted at the Open Science SE private beta, due to be closed tomorrow.

  • 4
    Again, this question seems to actually be a springboard for discussion on the topic of perpetuating reproducibility. I apologize for being a pain in the backside... I'm honestly impressed with the effort you guys are putting in to make this site merge work. Can this also be edited to make it more of an answerable question as opposed to an invitation for discussion? – eykanal Aug 20 '15 at 21:58
  • The first and only answer I got over there was very productive and to the point - hope it will be posted here too. In order to comply better with scope limitations, it would be helpful if you (or anyone) could point me to a place where scope - or criteria for holding or closing or protecting or deleting questions or answers or comments - are spelled out. Haven't gone through all of meta.academia.stackexchange.com/search?q=scope yet. – Daniel Mietchen Aug 20 '15 at 22:18
  • The original thread is now available through github.com/Daniel-Mietchen/SE-Static/blob/master/openscience/… , and I have found some help pages. – Daniel Mietchen Aug 20 '15 at 23:09
  • I agree with eykanal. I think there is an interesting question in here, but it needs to be more clearly articulated. What aspect of reproducibility are you asking about? How are you defining perpetuation? is it more than just having an effect of increasing reproducibility? – Jeromy Anglim Aug 21 '15 at 7:01
  • the common practice of framing plagiarism primarily or even exclusively as a copyright issue – Are you talking about academic plagiarism or plagiarism in general here? In the former case, I am not aware of a single case supporting this statement. – Wrzlprmft Aug 21 '15 at 7:03
5

A few ways to perpetuate reproducibility:

  1. As a stipulation to publication, publishers could require the software and data used for a research paper to be made publicly available so that others could reproduce the results.
  2. Academic institutions could provide repositories to allow its members to publish their code or data, to encourage making the code or data publicly available.

Difficulties regarding reproducibility requirements:

A major downside--perhaps a dealbreaker--is that there are significant issues with intellectual property. Large numerical models often take years to develop, benchmark and test, but once they are completed, they can often be used several times for several publications with minimal modification. Sharing the code they spent years developing in many cases would allow others to use the code to make discoveries before the original developers had a chance to do so. Allowing the original developers to protect their code gives them incentive to develop it.

Many data sets (especially in the social sciences) are protected/restricted because they contain classified, confidential, or personally-identifiable information. Moreover, many data sets that contain no personally identifiable information can actually be de-anonymized though careful computer analysis, so even protected data sets that have been "anonymized" are not necessarily safe to be released to the public.

Another difficulty is that some codes/scripts/routines may use commercial software (e.g. MATLAB, SPSS, etc.), so while the code themselves may be freely available, they might not necessarily be usable without paid software. This would exclude certain people from being able to reproduce results.

Some data sets are so large as to be an unreasonable burden on repositories. For example, I collaborated on a simulation that produced over 50 TB of data -- and there are other data sets that make 50 TB look tiny! It could cost a lot to maintain repositories on the order of hundreds of TB or several PB.

Summary

There are a few things that could be done to strongly encourage reproducibility, but they come at a cost. Releasing code, exact methodology, data sets, and other intellectual property would impact different fields in different ways and may not always be possible from a legal perspective. My guess is that data reproducibility will need to be treated on a field-by-field basis.

  • Your points 1 and 2 are key elements of reproducibility policies, and the issues you describe are important and frequently stand in the way of reproducibility. I also agree that field specificities need to be taken into account. However, my question is not about that, but about how we can make reproducibility viral, i.e. if I share my stuff reproducibly and you want to make use of it, you should share reproducibly too. – Daniel Mietchen Aug 27 '15 at 0:38
  • @DanielMietchen: Gotcha. Thanks. I'm not sure about that last bit. – jvriesem Aug 27 '15 at 13:13
2

It's an economic problem, not a legal problem. Make the experiment cheap to do, and it will be widely copied. Provide funding or career advancement for reproduction, and people will do it. An unfunded legal mandate will just cause scientists to go elsewhere.

  • It's not just economics, though: some problems are inherently complicated, limited to certain researchers, are expensive, or are just plain difficult so that they cannot be reproduced easily. I agree with your last statement for sure, though. – jvriesem Sep 16 '15 at 3:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.