26

Recently, we had a lecture about Reproducible research on of the slides was:

  • Reproducibility: start from the same samples/data, use the same methods, get the same results.

  • Replicability: conduct again the experiment with independent samples and/or methods to get confirmatory results.

Replicability = Reproducibility + Conduct experiment again

  • Replicability might be challenging in epidemiology (recruit again a cohort) or molecular biology (complex cell manipulation).

  • Reproducibility should be a minimum standard. One should strive to at least make his/her own research reproducible.

I find many articles that use software and when they hint what kind of analysis did they don't provide the code nor the data. There are public free ways of storing data and code of studies, and even with a DOI.

So far, I thought that some data might not be freely available because it has some private information (my field is bioinformatics), or the authors intend to use it for further investigations and want to keep for themselves the data.

The same happens with the code is the intellectual property of the lab or principal investigator, but retaining the rights of the code don't goes against the replicability.

Why are these papers accepted and publicized, even if they don't allow reproducibility?

Related: Reproducible Studies? and an example of the problem it causes Can up to 70% of scientific studies not be reproduced?.

Edit:
Some other papers about replicability: 1, 2.

  • Excel case: in this paper we can see an example. Reviewers of Growth in a Time of Debt estimated (note that this is not measurable/checkable afirmation) that the analysis could bring the results presented.

    (I couldn't find a description of the methods used for the analysis on the paper, but it is a different field of mine and I have skimmed through).
    But in new methods without prior experience/validation how can one estimate it without looking to the analysis itself?
    And in "old" methods, how bad would be to share them if they are already checked?

  • "Understand why replicability is important, and you'll understand which guidelines and rules should be applied, and how to deal with research where guidelines are skipped.":

    Replicability is important because science is about finding objective mesurable relations. This makes the relationship independent of who performs the study. But this can be discussed/answered on another question :)

    I am aware that we do our mistakes, (see my other question here on academia), but we should aim for the best behavior and the best science.

  • "Put another way, there are finite resources so the more you rerun the same code the less scientific progress you make overall"

    I don't think that we make less scientific progress overall rerunning the same code. Checking that we know for sure that A is true is far better than work for 3 years or more and then discover that A was wrong.
    How many finite resources are/were used on studies based on those the ALS Therapy Development Institute couldn't replicate?
    At the same time the induced pluripotent cells were hard to reproduce and replicate, but this isn't a software based, or the recent example of @tpg2114, 3 years to replicate their own study in 4 new settings.

  • Quality of academic software when sharing it, here it seems that it is better to share the awful, crapy code rather than hide it.

  • Necessity of the reviewer of the code and data was answered here. In short:

    Of course, the degree to which a referee is expected to verify the correctness of results varies greatly between fields. But you can always choose a personal standard higher than what's usual in your field. Just realize that good refereeing takes a significant time investment.

It seems that the comments are extending, and from the answers it seems that it is not the job of the reviewers or the journals, it might be a job for the reader (and of no one), or it doesn't worth it to make articles reproducible because it is too hard.

  • 1
    @101010111100 In my field, and fields close to mine, (almost) everybody uses Excel for data analysis. – Roland Jun 1 '16 at 11:46
  • 4
    @Roland I don't see how Excel would prevent people from sharing their data. – Cape Code Jun 1 '16 at 12:08
  • 2
    @CapeCode It doesn't. However, the data analysis is usually not reproducible (even by the person that did the analysis). – Roland Jun 1 '16 at 12:13
  • 13
    Reproducibility and replicability are important. The unthinking cult worshipping the trappings of reproducibility, though, is not. People asking "Why are this papers accepted and publicized even if they don't allow reproducibility?", or who mindlessly reject anything to do with Excel out of a sense of smug superiority, usually don't actually understand what they're asking about, but are looking at superficial aspects instead of actual data. This is cargo cult thinking, and is going to do more to harm real replicability movements than help. – iayork Jun 1 '16 at 12:53
  • 5
    I feel your question is fine. The comment made by @iayork seems overly sensitive and is reminiscient more of a misguided rant than a worthwhile contribution. Anybody who asserts that people asking a perfectly legitimate question belong to some "cargo cult thinking" club is probably best left ignored. – Jin Jun 1 '16 at 17:39
17

There are multiple answers to your question.

  1. Academic publishing assumes good faith. At least in the circles where the majority of people are genuinely interested in the advancement of science. This is the only viable mode of sharing scientific information between adults, although it sometimes fail as we all know it. Publishing should firstly serve the purpose of communicating results in a fast and convenient manner and second provide some sort of quality control.
  2. Sharing code and data is not a necessary condition for a study to be reproducible. It surely lowers the threshold of work that has to be done by other groups to reproduce the results, but clear and comprehensive descriptions of the methods and algorithm, as well as how the data was gathered, suffice. In fact, it's even better since starting from scratch will avoid the reproduction of results that are due to artifacts in the implementation or the original data.
  3. Acceptance standards vary vastly between journals. Nowadays anything can be "accepted and publicized" providing you pay the "article processing charge". The question is where was it published and by whom. The most prestigious journals in my field have strict guidelines about the study design and sharing of materials and methods, the other ones don't.
  • 14
    @Llopis Peer review is fundamentally not about catching cheaters. The basic Modus Operandi of peer review is (1) author describes faithfully what (s)he did and what results have been obtained (2) reviewer judges the validity of this method and the interestingness and relevance of the results. Basically, peer review does not work without good faith (on both sides). – xLeitix Jun 1 '16 at 12:39
  • 2
    @Llopis If you as a reviewer would start from a position of deep distrust, every review would essentially need a full replication study. – xLeitix Jun 1 '16 at 12:40
  • 2
    @Llopis If you review at least 5-10 papers a year (which is not much for an average researcher) would you be able to install 5-10 code sources and datasets on your PC and run them to see if the author is correct? And then repeat the statistical analysis the authors do? And then repeat again after a major revision? This is simply not feasible – Alexandros Jun 1 '16 at 13:28
  • 2
    @Llopis "I doubt the code would take longer than 1 day to run" I don't think so nature.com/news/… – d.putto Jun 1 '16 at 14:53
  • 4
    @Llopis if you're reviewing and find a method to be sketchy to the point that you doubt it can produce the presented results, then by all means ask for clarifications or the code. If you find the methods to be sound ans appropriate and you're passionate about the subject and you have the time and resources to put it to the test, go ahead and publish your replication or contradictory study. – Cape Code Jun 1 '16 at 15:08
9

Sometimes reproducing results is as much work as producing them. If I give you my code, you not only have to check whether it gives the results I published, but also whether it is correct. Reading poorly written code is a pain, so it might be easier for you to write it in your style again. Also I might have used a language like Fortran or some obscure computer algebra system.

The same is true for certain computations, in particular if they involve a lot of case distinctions, or if there are many different ways to obtain the result.

So I would leave the reproduction to the reader if I have the feeling that my guidance would not be helpful. The problem with this view is of course that it is open to rationalizations: Writing the code in a reader friendly way is too much work, so I invent a reason not to have to do it. Preventing this would be the job of the reviewer, but why should they be less lazy than me?

  • 2
    +1 In my experience, the main reason people don't share their code is because said code is poorly written. – 101010111100 Jun 1 '16 at 11:48
  • 1
    Even poorly code is better than no code in my opinion. At least one get some sense of the functions one use or how they are used anf if the results are obtained this way. This is enterly different from discussing the methods which would requiere in depth knowledge of the code – llrs Jun 1 '16 at 12:08
  • 1
    @Llopis: I have listed some possible practical obstacles to publishing one's code in another question (several of which could be called facets of "poorly written"). – O. R. Mapper Jun 1 '16 at 14:31
  • 3
    @Llopis No, actually providing bad code hurts anyone who wants to replicate. Just running the same code is not replicating or reproducing results - it's more like watching a video of the experiment. Sure, it can point out a blatant mistake, but in general - you still want to do it yourself to be sure if you're skeptical. And code can have subtle flaws that are not part of the method (but rather part of the implementation). No amount of eyeballing is going to catch that - you either have to perform full analysis (and that's a huge task), or write it yourself... – Ordous Jun 1 '16 at 18:51
  • 1
    ...[cont] Unfortunately the human brain tends to replicate solutions it already saw. So if you are writing a similar thing yourself to test, you will invariably borrow ideas or constructions (and possibly flaws) from the code that you already read. This is something that has been known for centuries (although not applied to programming obviously). So unless you are talking about a paper where the actual implementation, rather than method matters (algorithmic real-world performance, for example), giving your bad code along with your paper does not help, but rather hurts. – Ordous Jun 1 '16 at 18:51
2

On top of the existing answers regarding the current state of reproducibility in peer review, it is important to consider that norms regarding reproducibility are rapidly changing in peer review and publication among:

Peer reviewers

The Center for Open Science endorses a standard disclosure request that you can add to any peer review to request that the authors provide minimal information on reproducibility:

I request that the authors add a statement to the paper confirming whether, for all experiments, they have reported all measures, conditions, data exclusions, and how they determined their sample sizes. The authors should, of course, add any additional text to ensure the statement is accurate. This is the standard reviewer disclosure request endorsed by the Center for Open Science [see http://osf.io/hadz3]. I include it in every review.

I use this myself and have found that at least a handful of journal editors will press the authors on this front.

Journal Editors

  • Several top journals in psychology are leading the way in terms of incentivizing replication-friendly publications by providing "badges" indicating that the paper has released raw data, had its protocol preregistered, and/or has released study materials. Recent empirical evidence has suggested that such incentives are effective.
  • Numerous journals now require a statement about whether data are publicly available, or an explanation of why they cannot be made so (e.g., due to sensitive information). There are too many to list, but here's a commendable example from Science:

After publication, all data and materials necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. After publication, all reasonable requests for data or materials must be fulfilled.

Authors

Even when submitting to a journal that does not have any requirements regarding reproducibility, some authors are choosing to make this information conspicuous anyway. On my own papers, I now add a "Research Transparency" statement before the Acknowledgments, saying simply:

All raw data, materials, and R code are publicly available at [my Open Science Framework repository URL].

I also say something to this effect in my cover letters when submitting papers. These voluntary disclosures may help promote new community norms, hopefully eventually making poor disclosure a red flag rather than the status quo. In fact, I experienced a small example of this when a reviewer recently commented:

I appreciate the authors [sic] releasing all their data and protocols online.

1

It's often technically difficult to recreate the same runtime as another person used, so the code may not do the same thing on your machine. In practice, we're usually not talking about tested, portable, production quality software. You may need a certain operating system, certain libraries to be installed, the correct compiler or interpreter version etc.

Reproducible computation is something the SciPy community was deeply interested in a few years ago, and seemed close to getting a handle on. I'm not sure how far they've come since (I'm not a scientist, so it was off my radar), but I know it was an important issue for IPython Dev, and Continuum Analytics were working on it as well.

You need to share the whole setup, maybe as a machine image for Amazon Web Services, or use Docker or something like that. You basically do your work in VMs, then share whatever people need to reproduce the VM.

  • Yes, I am aware of such solutions. Usually Docker is used to catch the environment where software is developed and tested in industry. Precisely because such technical solutions of using a fixed environment and the analysis (Ipython for python, sweave and knitr in R,...) exists I am concerned about not sharing neither the data nor the code, which under my point of view would be the minimum. However I am afraid your insights don't answer the question. – llrs Jun 2 '16 at 7:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.