Open Source Software and Reproducibility in Computer Science

Question

As a PhD researcher in computer science, my scientific results often take the form of algorithms, which in turn come with implementations. I am convinced that releasing these implementations under an open-source model has various benefits especially for scientific codes, one of which is reproducibility of results. I also believe that "Release Often, Release Early" applies to science, as it opens up the possibility of feedback that ultimately leads to better software and better research.

However, my advisor is concerned with the danger of plagiarism in case the code is released before the respective paper is published, and therefore disagrees with that philosophy. To me it seems like a very theoretical threat since I have never heard of a comparable case of plagiarism. However, we just got a paper rejected in which one major criticism of the reviewer was that it was impossible to reproduce results since the code was not available. My advisor thinks that this is unfair criticism because the conference has not explicitly asked for a release of the source code.

What do you think? Is it sensible to keep implementations secret until the paper is through, or is open-source actually a requirement for reproducible computer science?

Answer 1

Computer science is a particularly friendly environment for releasing material in advance of publication. In fact, there are a number of methods for doing so that provide a clear archival time-stamp on your work, including university technical reports, arXiv, and big repository sites like github and bitbucket. Moreover, unlike many other disciplines, such pre-release almost never impairs your ability to build conference and journal publications on top of your released material. In fact, you can typically release not just your code, but also your draft paper through the same mechanisms.

If you do such pre-release, you are essentially immune to plagiarism, because anybody who pirates your results will look like a fool with date-stamps marking their guilt for all to see. It may not stop bottom-feeders from trying, but it will mean that nobody you care about (including reviewers) should take such an attempt seriously.

The only likely problem that you can run into is double-blind reviewing, as discussed in this question. If that's not a major concern for you, or if you can use one of the mechanisms suggested in that question, then I see no likely downside in computer science publication.

On a side note, however: just because the code is available doesn't mean it's reproducible, as anybody who's spent significant time having a dependency-fight with poorly-document open source projects can attest.

Answer 2

I wish that more papers were rejected for such reasons, that more grant funding agencies required release, and that more publishing venues did so as well. That this is not the case, is a reality we have to live in.

A paper should be, at the very least, good enough that an intrepid reviewer could follow the description through to an implementation of their own that works and can reproduce its results. I have certainly seen this done for one of my own papers where our code was not ever published and another author compared their later algorithm to ours by reimplementing our method. In fact, I asked them to do this as part of the review process using their code for their problem since our two approaches were so similar. It made for a much better paper.

I don't believe that in most cases simply rerunning someone's code on the same inputs tells us much more than the paper does. If you don't believe that someone's implementation does what they says it does, you are essentially accusing them of fabricating their results. If this is your concern as a reviewer, then you should say so. E.g., "The results of this paper are unbelievably good! I have worked through the implications of their algorithm and cannot see how an implementation on a real computer could achieve this level of performance. The authors should provide (at least to this reviewer through the editor) code, inputs, and instructions sufficient to reproduce the results in Table 10." Now, they may tell you and the editor to go pound sand, but the peer review process is where this can be addressed now.

Additionally, there are other reasons to keep the code secret. The authors may in some stage of patenting their software. I think that software patents are wrong-headed, but the fact of the matter is that US law allows federal grant awardees to pursue patents for things they invent under federal grants (with a non-exclusive license for use back to the government). In some circumstances, the authors may wish to pursue a patent on the implementation while describing the methodology and mathematics in the open literature. This can be tricky.

Answer 3

Having made a career in open source software, I have two comments:

• I've never had a case where someone "scooped" me. My code has often been available in some form or other (on open software repositories) months or years before the paper was finally ready, but nobody ever scooped me on it. On the other hand, a fair share of my collaborators thought that the code I produced was good, do not have the requisite knowledge to work with it themselves, and asked to collaborate with me. All very positive.

• I've probably written 10-15 papers for which the source code is available (as open source, well documented, production-ready code). In almost all of these cases, paper reviewers thought that very positive and it helped the paper getting accepted.