I am submitting a paper to a machine learning/computer vision conference. Is it worthwhile to include some source code of my implementation so that the reviewers can test it themselves or gain a greater understanding of my idea? It will take time to clean up my code and make sure it is anonymised.

Is this a common practice? If so do reviewers pay attention to the code?

Is it worth the time to prepare anonymised and clean code?

The code will be included as a ZIP archive through the paper submission website. After the paper has been accepted I intend to release it with an open source license.

  • 1
    Could you clarify if the source code is to be in the paper/pdf submitted, or is a link to a repository where it's stored like github?
    – user389823
    Commented Oct 30, 2013 at 0:30
  • @user389823 It is to be suppled in a ZIP archive on the paper submission website. Commented Oct 30, 2013 at 3:44
  • 1
    related, but not a dupe academia.stackexchange.com/questions/10629/…
    – user7130
    Commented Oct 30, 2013 at 8:41
  • What do you mean by anonymized? Are you anonymizing yourself (and the other code authors) or the subjects of the study? Commented Nov 5, 2013 at 5:46
  • @DavidRobinson The former. Commented Nov 5, 2013 at 12:41

4 Answers 4


Should I supply code as supplemental material?

Yes, but it would be better if you published it in an open access fashion.

Is this a common practice?


If so do reviewers pay attention to the code?

Depends on the reviewers.

Is it worth the time to prepare anonymised and clean code?

That depends, don't do more than necessary.

Also read:

  • 1
    I plan to release it under an open source license once the paper is accepted somewhere. Commented Oct 30, 2013 at 3:49
  • @StephenTierney I may have misunderstood the question. In this case I'd suggest to protect your work by publishing it. In some areas it is customary to send the paper to arxiv.org or similar repositories before sending it for review, I don't know how trustworthy may be the reviewers in this specific case you have. IMHO the design of the whole review-publish process is wrong, but that's a different matter.
    – Trylks
    Commented Oct 30, 2013 at 4:13
  • 1
    @StephenTierney why don't you just publish the source code right now, let's say on Github.com, under an open source license? You would protect it before submission, and it would run away from copyright transfer agreements.
    – user7112
    Commented Feb 11, 2014 at 12:30

Making code available is always a good idea, as long as the code works. It is probably a good idea even if it doesn't, since even non-working code can help reviewers and users/readers of your work to understand details of your implementation that your paper does not cover. However, if the code does not work, you should clearly indicate this in the source. In the case of working and tested code, bear in mind that sufficient documentation to run the code is highly desirable. You only mention reviewers, but you should also be thinking about general readers of your paper.

I think it is probably reasonable to supply a current snapshot of the code to the paper as a zip archive (or similar) for reviewing purposes, but why not just put it online directly as a Git or Mercurial repository on Bitbucket, Github, or similar, and reference this on the paper? I also recommend making repositories available in more than one place, in the interests of redundancy. For example, I have used both Bitbucket and Google Code for my Mercurial repositories. This has various advantages over a zip archive file; for one thing you can push corrections and other changes to your repository, and everyone will immediately have access to them.

If you are concerned about releasing your code before your paper has been published might mean someone else will "scoop" you, that seems unlikely to me. At least, it is not something I've ever worried about.

  • 1
    I am only concerned with the impact supplying code has on the review process. I cannot supply the code on an external website as it may compromise anonymity in the review process. Commented Nov 1, 2013 at 9:57
  • @StephenTierney I understand your point. However, you seem to be referring to a situation where the reviewers do not know the author's identity. However, my impression is that this is rarely the case in practice, because such a restriction is difficult to enforce. For example, how does one handle the case where the author refers to his/her previous works in the bibliography? Commented Nov 1, 2013 at 12:34
  • When you refer to your previous work you say "Tierney et al. proposed" rather than "We previously proposed". Commented Nov 2, 2013 at 4:30
  • True, but if the paper in question depends on earlier publications by the author, and those earlier publications are in the bibliography and not anonymized, then often the current authorship is quite clear, if only from style/content considerations. Anyway, enough said. Commented Nov 2, 2013 at 11:46

Absolutely, yes.

If you don't release code, then your work is not de-facto reproducible; and if you will get me as a reviewer, I will do everything in my power to either get you to publish code, or get rejected. Rare exceptions to the contrary, where an implementation is trivial. But implementations are rarely trivial; and often a great place to sweep a lot of quirks under the rug which conveniently arnt mention in the paper.

Granted, it is not hard to get away with not providing code; journals are particularly lax at enforcing or even adopting standards, even though most people in the field will decry the lack of reproducibility of published work. But nobody will resent you for providing code either. Writing code you actually feel comfortable releasing out into the open requires work of course. Time which you could spend doing other things. If you don't feel comfortable zipping up your code and providing it right now, that's probably a good indication that any papers written about said code arnt ready to be published yet either.

Unfortunately, there is a tension there, between doing what is right scientifically, and doing what is right for your career. But as for me as a reviewer; no code == no publication.

  • 1
    Love it, love everything about it! Although I'd still say, hideous code can still make a huge impact. So if it's clean code vs hideous code.... then yea clean the code. If it's hideous code vs sans-code, no question still publish code. Commented Jun 14, 2014 at 5:42

This largely depends upon the reviewers and the field customs. Some reviewers, specially in journals, may ask for source code. However, it is unlikely that in Computer Science conferences reviewers ask for source code specially when the paper is well-written.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .