I am submitting a paper to a machine learning/computer vision conference. Is it worthwhile to include some source code of my implementation so that the reviewers can test it themselves or gain a greater understanding of my idea? It will take time to clean up my code and make sure it is anonymised.

Is this a common practice? If so do reviewers pay attention to the code?

Is it worth the time to prepare anonymised and clean code?

The code will be included as a ZIP archive through the paper submission website. After the paper has been accepted I intend to release it with an open source license.

  • 1
    Could you clarify if the source code is to be in the paper/pdf submitted, or is a link to a repository where it's stored like github? – user389823 Oct 30 '13 at 0:30
  • @user389823 It is to be suppled in a ZIP archive on the paper submission website. – Stephen Tierney Oct 30 '13 at 3:44
  • 1
    related, but not a dupe academia.stackexchange.com/questions/10629/… – user7130 Oct 30 '13 at 8:41
  • What do you mean by anonymized? Are you anonymizing yourself (and the other code authors) or the subjects of the study? – David Robinson Nov 5 '13 at 5:46
  • @DavidRobinson The former. – Stephen Tierney Nov 5 '13 at 12:41

Should I supply code as supplemental material?

Yes, but it would be better if you published it in an open access fashion.

Is this a common practice?


If so do reviewers pay attention to the code?

Depends on the reviewers.

Is it worth the time to prepare anonymised and clean code?

That depends, don't do more than necessary.

Also read:

  • 1
    I plan to release it under an open source license once the paper is accepted somewhere. – Stephen Tierney Oct 30 '13 at 3:49
  • @StephenTierney I may have misunderstood the question. In this case I'd suggest to protect your work by publishing it. In some areas it is customary to send the paper to arxiv.org or similar repositories before sending it for review, I don't know how trustworthy may be the reviewers in this specific case you have. IMHO the design of the whole review-publish process is wrong, but that's a different matter. – Trylks Oct 30 '13 at 4:13
  • 1
    @StephenTierney why don't you just publish the source code right now, let's say on Github.com, under an open source license? You would protect it before submission, and it would run away from copyright transfer agreements. – user7112 Feb 11 '14 at 12:30

Making code available is always a good idea, as long as the code works. It is probably a good idea even if it doesn't, since even non-working code can help reviewers and users/readers of your work to understand details of your implementation that your paper does not cover. However, if the code does not work, you should clearly indicate this in the source. In the case of working and tested code, bear in mind that sufficient documentation to run the code is highly desirable. You only mention reviewers, but you should also be thinking about general readers of your paper.

I think it is probably reasonable to supply a current snapshot of the code to the paper as a zip archive (or similar) for reviewing purposes, but why not just put it online directly as a Git or Mercurial repository on Bitbucket, Github, or similar, and reference this on the paper? I also recommend making repositories available in more than one place, in the interests of redundancy. For example, I have used both Bitbucket and Google Code for my Mercurial repositories. This has various advantages over a zip archive file; for one thing you can push corrections and other changes to your repository, and everyone will immediately have access to them.

If you are concerned about releasing your code before your paper has been published might mean someone else will "scoop" you, that seems unlikely to me. At least, it is not something I've ever worried about.

  • 1
    I am only concerned with the impact supplying code has on the review process. I cannot supply the code on an external website as it may compromise anonymity in the review process. – Stephen Tierney Nov 1 '13 at 9:57
  • @StephenTierney I understand your point. However, you seem to be referring to a situation where the reviewers do not know the author's identity. However, my impression is that this is rarely the case in practice, because such a restriction is difficult to enforce. For example, how does one handle the case where the author refers to his/her previous works in the bibliography? – Faheem Mitha Nov 1 '13 at 12:34
  • When you refer to your previous work you say "Tierney et al. proposed" rather than "We previously proposed". – Stephen Tierney Nov 2 '13 at 4:30
  • True, but if the paper in question depends on earlier publications by the author, and those earlier publications are in the bibliography and not anonymized, then often the current authorship is quite clear, if only from style/content considerations. Anyway, enough said. – Faheem Mitha Nov 2 '13 at 11:46

Absolutely, yes.

If you don't release code, then your work is not de-facto reproducible; and if you will get me as a reviewer, I will do everything in my power to either get you to publish code, or get rejected. Rare exceptions to the contrary, where an implementation is trivial. But implementations are rarely trivial; and often a great place to sweep a lot of quirks under the rug which conveniently arnt mention in the paper.

Granted, it is not hard to get away with not providing code; journals are particularly lax at enforcing or even adopting standards, even though most people in the field will decry the lack of reproducibility of published work. But nobody will resent you for providing code either. Writing code you actually feel comfortable releasing out into the open requires work of course. Time which you could spend doing other things. If you don't feel comfortable zipping up your code and providing it right now, that's probably a good indication that any papers written about said code arnt ready to be published yet either.

Unfortunately, there is a tension there, between doing what is right scientifically, and doing what is right for your career. But as for me as a reviewer; no code == no publication.

  • 1
    Love it, love everything about it! Although I'd still say, hideous code can still make a huge impact. So if it's clean code vs hideous code.... then yea clean the code. If it's hideous code vs sans-code, no question still publish code. – J. M. Becker Jun 14 '14 at 5:42

This largely depends upon the reviewers and the field customs. Some reviewers, specially in journals, may ask for source code. However, it is unlikely that in Computer Science conferences reviewers ask for source code specially when the paper is well-written.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.