16

As a PhD researcher in computer science, my scientific results often take the form of algorithms, which in turn come with implementations. I am convinced that releasing these implementations under an open-source model has various benefits especially for scientific codes, one of which is reproducibility of results. I also believe that "Release Often, Release Early" applies to science, as it opens up the possibility of feedback that ultimately leads to better software and better research.

However, my advisor is concerned with the danger of plagiarism in case the code is released before the respective paper is published, and therefore disagrees with that philosophy. To me it seems like a very theoretical threat since I have never heard of a comparable case of plagiarism. However, we just got a paper rejected in which one major criticism of the reviewer was that it was impossible to reproduce results since the code was not available. My advisor thinks that this is unfair criticism because the conference has not explicitly asked for a release of the source code.

What do you think? Is it sensible to keep implementations secret until the paper is through, or is open-source actually a requirement for reproducible computer science?

Improve this question
6
  • 1
    (Open source) reference implementations aren't a strict requirement yet, but the faster they become one the better. I don't see how plagiarism is even relevant, as the source code isn't published as text.
    – Marc Claesen
    Jun 16, 2015 at 21:15
  • @MarcClaesen: Theoretical plagiarism workflow: Take the code from our release, run some experiments with it, claim work as your own, publish first, get credit.
    – clstaudt
    Jun 16, 2015 at 21:17
  • 13
    (1) Nobody will know the code well enough to get useful results that fast. (2) You need to know what's going on in the code to write a paper about it. (3) Your own paper should be on arXiv or a technical report, immediately proving you were first. (4) The use of your code should be acknowledged.
    – Marc Claesen
    Jun 16, 2015 at 21:19
  • 1
    Why not publish code and preprint at the same time, then submit?
    – Raphael
    Jun 17, 2015 at 10:10
  • I don't believe there's a single answer that will be right for all situations. I suspect this is highly dependent on the specific situation and context, and on the specific benefits for releasing early. In any case, this sounds like a call for opinion about a broad class of situations, without a lot of context to enable providing an informative answer.
    – D.W.
    Jun 17, 2015 at 19:08

3 Answers 3

Reset to default
27

Computer science is a particularly friendly environment for releasing material in advance of publication. In fact, there are a number of methods for doing so that provide a clear archival time-stamp on your work, including university technical reports, arXiv, and big repository sites like github and bitbucket. Moreover, unlike many other disciplines, such pre-release almost never impairs your ability to build conference and journal publications on top of your released material. In fact, you can typically release not just your code, but also your draft paper through the same mechanisms.

If you do such pre-release, you are essentially immune to plagiarism, because anybody who pirates your results will look like a fool with date-stamps marking their guilt for all to see. It may not stop bottom-feeders from trying, but it will mean that nobody you care about (including reviewers) should take such an attempt seriously.

The only likely problem that you can run into is double-blind reviewing, as discussed in this question. If that's not a major concern for you, or if you can use one of the mechanisms suggested in that question, then I see no likely downside in computer science publication.

On a side note, however: just because the code is available doesn't mean it's reproducible, as anybody who's spent significant time having a dependency-fight with poorly-document open source projects can attest.

Improve this answer
4
  • 2
    Remember that timestamps are significant in arXiv, but certainly not in Git, where it is trivial to rewrite history a posteriori.
    – Clément
    Jun 17, 2015 at 8:02
  • 2
    And if the code is hosted in Bitbucket and it can be made private (for free), shared with the reviewers, and then made public after the publication. Or so I think.
    – fridaymeetssunday
    Jun 17, 2015 at 10:04
  • 1
    @Clément There seem to be ways around that with the added benefits of getting you a DOI for your code. I'm not sure this is the way for a preprint (which may yet evolve), but for the "real" publication one should probably do this.
    – Raphael
    Jun 17, 2015 at 10:12
  • 1
    It should be noted that directly working on a public repository such as on Github automatically makes commit times visible, but can actually give away too much information, as I have also described in another answer. These particular issues can mostly be mitigated by only pushing certain pre-releases to the repository rather than every single commit.
    – O. R. Mapper
    Jun 17, 2015 at 13:39
11

I wish that more papers were rejected for such reasons, that more grant funding agencies required release, and that more publishing venues did so as well. That this is not the case, is a reality we have to live in.

A paper should be, at the very least, good enough that an intrepid reviewer could follow the description through to an implementation of their own that works and can reproduce its results. I have certainly seen this done for one of my own papers where our code was not ever published and another author compared their later algorithm to ours by reimplementing our method. In fact, I asked them to do this as part of the review process using their code for their problem since our two approaches were so similar. It made for a much better paper.

I don't believe that in most cases simply rerunning someone's code on the same inputs tells us much more than the paper does. If you don't believe that someone's implementation does what they says it does, you are essentially accusing them of fabricating their results. If this is your concern as a reviewer, then you should say so. E.g., "The results of this paper are unbelievably good! I have worked through the implications of their algorithm and cannot see how an implementation on a real computer could achieve this level of performance. The authors should provide (at least to this reviewer through the editor) code, inputs, and instructions sufficient to reproduce the results in Table 10." Now, they may tell you and the editor to go pound sand, but the peer review process is where this can be addressed now.

Additionally, there are other reasons to keep the code secret. The authors may in some stage of patenting their software. I think that software patents are wrong-headed, but the fact of the matter is that US law allows federal grant awardees to pursue patents for things they invent under federal grants (with a non-exclusive license for use back to the government). In some circumstances, the authors may wish to pursue a patent on the implementation while describing the methodology and mathematics in the open literature. This can be tricky.

Improve this answer
3
  • So was the reviewer justified in taking this as one reason for rejection? On the one hand you seem to think so, on the other hand you say that the reviewer should state it more openly if she has doubts about the results.
    – clstaudt
    Jun 16, 2015 at 21:39
  • 1
    Being unsure about the truthfulness of results or whether reported performance generalizes to other problems is a very good reason for rejection. Providing code as supplementary material is almost always possible, and pretty much entirely addresses such concerns. I'd be happy if major computer science journals/conferences would require reference implementations.
    – Marc Claesen
    Jun 16, 2015 at 21:44
  • 1
    @cls, I think it's justifiable, but it's also a matter of opinion that's yet unsettled in many scientific communities. There are at least two actors here. If this was the only reason the reviewer gave, and the journal editor accepted it, I'd have some concerns for the editor's behavior, too. At the very least, I would have written a response to the editor offering to share the code (though the editor) with the reviewer (at the least) to help alleviate their concerns. If there were 15 other reasons given for rejection, maybe you have to take your lumps, revise the paper, and submit elsewhere.
    – Bill Barth
    Jun 16, 2015 at 23:14
10

Having made a career in open source software, I have two comments:

  • I've never had a case where someone "scooped" me. My code has often been available in some form or other (on open software repositories) months or years before the paper was finally ready, but nobody ever scooped me on it. On the other hand, a fair share of my collaborators thought that the code I produced was good, do not have the requisite knowledge to work with it themselves, and asked to collaborate with me. All very positive.

  • I've probably written 10-15 papers for which the source code is available (as open source, well documented, production-ready code). In almost all of these cases, paper reviewers thought that very positive and it helped the paper getting accepted.

Improve this answer
2
  • This is a great example. Do you have any evidence that a reviewer actually reran anything in one of your articles. I have the feeling that reviewers in PDE simulation sometimes play lip service to reproducibility by wanting to see code, but I haven't ever seen anyone try to actually reproduce anything. You're also a journal editor, so maybe you have some examples?
    – Bill Barth
    Jun 17, 2015 at 15:34
  • 3
    No, I have no evidence whatsoever that reviewers ever ran any of my codes during review. But that's beside the point -- they thought it was a service to the readers of my papers to make the code available, and that increased the relevance score you always have in the back of your mind as a reviewer.
    – Wolfgang Bangerth
    Jun 17, 2015 at 17:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .