21

When publishing a paper, some researchers publish the source code used for the paper.

Is there any research/study/survey/... that looked at how much effort do researchers take to publish their source code? I.e. how many hours do researchers take to publish the source code for a given paper?

18

This isn't a peer-reviewed article, but nonetheless it's worth linking to because it specifically addresses your question, albeit as an n=1 case:

Bruna, E. 2014 THE OPPORTUNITY COST OF MY #OPENSCIENCE WAS 36 HOURS + $690

In this blog post, a biologist Emilio Bruna states it took about 25 hours of his time to appropriately document his code (associated with a research paper) to make it good enough for open source release on github.

  • 3
    Amount of money is not for code, but for an open science publication ($600) and data storage ($90). – Piotr Migdal May 3 '15 at 22:20
  • 4
    And you can archive your data and code for free (e.g., Figshare) and you can share source code and data without publishing in a journal that requires payment to publish. – Jeromy Anglim May 4 '15 at 1:33
  • 3
    Documentation is commendable, but probably not strictly necessary to qualify for the "open" label. May depend on the code. – Raphael May 4 '15 at 10:27
  • 1
    I fail to find the problem with undocumented code in research, and not having code documented should not be a barrier to releasing it for others. Sure, if your research is building a large sophisticated model that you plan on continuing to develop (especially if you have others working on it), then it should be documented. But some R code for analysis of bioinformatics probably doesn't need sophisticated documentation. That said, there is no requirement for code to be documented in order to be open-source. It's really just a courtesy for others, if it's not being developed in industry. – Chris Cirefice May 4 '15 at 15:05
9

This is a study that analyzes whether computer science papers include source code that makes it easy to reproduce their results.

http://reproducibility.cs.arizona.edu/

The study found that out of 601 papers analyzed, 139 included source code that could be obtained without contacting the authors, and the study's researchers were able to email authors to get the source code for an additional 87 papers.

Of the 226 papers the authors obtained source code for, they were able to configure and run the source code within half an hour on 130 papers, without contacting the authors on an additional 64 papers, and after contacting the authors on a further 23 papers. For 9 papers, the study's researchers could not run the source code at all.

These results don't show how much time researchers spend on making their source code available, but it does show how frequently papers are published with accompanying source code and what quality that source code tends to be.

PLL made an excellent comment. I'd like to add it to my answer in case it disappears later:

Just to summarise: the overall success rate should be seen as 217 out of 402. Of the full sample of 608, 206 were excluded for some reason or another --- e.g. their results weren't based on code in the first place. 402 were left that should have contained code.

  • 4
    Just to summarise: the overall success rate should be seen as 217 out of 402. Of the full sample of 608, 206 were excluded for some reason or another --- e.g. their results weren't based on code in the first place. 402 were left that should have contained code. – PLL May 4 '15 at 9:12
  • 1
    Means that half of the papers that do rely on code should not have been published according to scientific standards. Go figure. – Raphael May 4 '15 at 10:28
  • 1
    A more recent article, focused on Zenodo: frontiersin.org/articles/10.3389/frma.2017.00013/full – Nemo May 2 '18 at 13:07
7

I have a significant number of papers where we make the (nontrivial) code available through various means. Examples are here:

In all of these cases, documenting the code adequately to make it suitable for publication along with the paper was part of writing the code (like for all significant code, documenting should be part of writing it) and would likely have taken at least 2 days in the case of the tutorials, and maybe 4-8 hours in the case of some of the specific codes and algorithms. It's something one should do anyway, but even if one doesn't, doing it is not usually an overwhelming effort.

Of course, this would not apply to a code like ASPECT for which writing the documentation (such as the 230 page manual) is an effort that likely represents month of work.

1

I disagree with your premise that it actually takes any effort at all.

Version control is needed anyways for collaboration with your coauthors, as backup and for version history, which GitHub, BitBucket and SourceForge offer for free.

Writing good code is necessary so hat your colleagues can understand what you are doing and even if it is a one-person project you need to understand it half a year later.

There are even additional benefits:

  • increased acceptance chance of publications
  • bug reports through issue tracking help with development
  • increased exposure and more citations

So you only hurt yourself if you don't publish your source code which means there is actually negative effort, all things considered.

  • 2
    "Version control is needed anyways for collaboration with your coauthors" -- I wish. Some refuse to use such, leaving some member to incorporate diffs, or do all the work on their own. "Writing good code is necessary [...]" -- should be; oftentimes you see "that's future us' problem". I don't think anybody would disagree with you on that versioning and good code hygiene would be preferable, but many make do without. For better or worse. – Raphael May 4 '15 at 10:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.