When should one publish code publicly on GitHub: during or after journal review?

Question

Similar to the following question, I have developed a code to produce results in a manuscript that I've written which I wish to now share with the community. But this manuscript, which is currently on arXiv, is still under journal review (which has taken several months and still sits with the referees). Therefore, I was wondering if I should put the code publicly on Github while the manuscript remains under review or only after the manuscript has been fully accepted/published by a journal? Does the right answer depend on the journal? Is it better to wait for any reason(s)? Any and all insights are welcome.

Answer 1

Particular to your situation, I think the exact answer is highly dependent on some missing information:

• How do you plan to license the code once released?
• Do you have some sort of agreement with the journal that the code is part of the manuscript that they will own the copyright to?

In General: When should one release the code relative to publication? Before! (Allow me to explain)

My knee-jerk response was "why isn't the code already on GitHub (or another hosting site)"? By publicly posting the code, publicly affiliating yourselves to it, and having a time-stamped "paper" trail of commits. I would expect this provides a good hedge against being "scooped", similar to, but more robust than, the old signed-and-dated lab books for maintaining development records in the case of IP/patent disputes. I work, directly or indirectly (and often in a very minor way) on computational chemistry code, but we write the code and store it in a GitHub repo where it's public facing and has open source licenses. The code either gets cited in papers that use it (not necessarily well - see this question) and/or there's an article written up detailing the software that provide details specific to it (and maybe even later publications of new major versions with added features, e.g. RMG and RMG 3 with actively-developed source here)

Answer 2

You should post it before the paper is sent for review. Your paper is already on arXiv, and you do plan to make the code public after publication, so I see little reason to be secretive with it just for the duration of the review.

With certain types of papers, the review can be much more productive (and also easier for the reviewer) if the reviewer has access to the code. In some cases, having access to the code is critical for the review.

Answer 3

The concerns to weigh are:

• Scooping. Your manuscript is already out on Arxiv, so there is no risk that someone will publish your same result earlier than you. By publishing code, you make it easier for other researchers to publish follow-up works. Is this something you want? Generally yes: more science, more reproducibility, more common world knowledge, and, more selfishly, more reputation in the community and more citations for you! But some prefer to keep their cards close to their chest, especially if they are already working on a natural follow-up.
• Anonymity. Is this a double-blind submission where you wish to hide your identity? I suppose not, in your case, since there is a preprint already out on Arxiv, so you are already non-anonymous.
• Replication. The referees may want to try your code and replicate your results. In some journals/areas, the code is considered a part of your paper, so if that is egregiously badly written that would reflect badly on your manuscript.
• License. Some journals forbid you to publish post-prints, because they already incorporate their 'work' (that referees and editors did for free for them). In theory, this could extend to code, so it is better to get it out early, when they can make no claim. I am not sure if it is even legal (and I am not sure if publishers ever sued anyone to enforce that clause or sent out DMCA requests to Github), but if you want the publishers to keep their greedy hands off your code it's better to publish it before submission (and the same applies to preprints).

The standards on all these points vary in different fields, and individual researchers have different opinions, so it is difficult to give a unique catch-all answer. At least in my field, though, sharing code is considered a good thing and I encourage you to do it.

Answer 4

I frequently end up reviewing papers where code is a portion of the results, and in that case if the code isn't provided I'll ask for it while reviewing the other parts of the paper that I can review. If the paper is entirely about a computational method and the code isn't included I turn it right back around to the editor with "Unreviewable without the code". I think this is necessary, because I would say that 90% of my review for these types of papers can be summarized as "Write a better methods section and write tests for your code".

I'm going to add one thing to the other answers (which are all quite good). It's very helpful to me to have a versioned release on GitHub (or PyPi or CRAN or conda or Zenodo, or really whatever release channel you want) that's the exact version that you used for your analysis. I do occasionally have the desire to see how a specific thing was implemented, and if it's buried in 700 commits because you're still working on your software (which is totally fine), it's really hard to figure out where your results came from.