In recent years there has been a drive to make science more open. This includes making the software used to perform research open source. The main argument in favour of this idea is that research should be reproducible, which has been addressed in other questions on this site.

I am more interested in the 'impact' that this produces. In particular, is there evidence that by publishing software alongside research papers means that more people use the methods described in the paper?

I am especially concerned with papers in applied mathematics but also more generally.

  • 1
    It may be worth pointing out that the impact you seem to describe (people using a presented method for whatever they do, i.e. application of research results) is not the impact aimed at based on your description in the preceding paragraph (people re-running an experiment to verify statements made in a paper, i.e. verification of research results). – O. R. Mapper Feb 5 '15 at 14:32
up vote 9 down vote accepted

You ask "In particular, is there evidence that by publishing software alongside research papers means that more people use the methods described in the paper?"

I am not aware of any large scale empirical studies that have assessed this for software at present [1] however there is anecdotal evidence and related studies looking at the impact of open data.

One piece of related anecdotal evidence came from short survey I carried out in 2013 looking for people who had been "scooped" as a result of publishing their code openly (i.e. someone else had published a paper using that software to get to the same scientific results before the author of the software). Whilst there were very few examples of people getting scooped, there were many more examples of researchers who had gotten new collaborations, new citations, and new funding as a result of publishing their code openly. Many said that this was because others were more able to try out the methods because code was available.

More convincingly, there have been several good studies looking at the effects on citations (a proxy for the sort of impact you mention) from making data openly accessible [2-6]. Many of the reasons for the data citation benefit discussed in [7] appear qualitatively to be true for software as well [8-9].

Finally, in the area of Applied Mathematics, you might like to look at some of the outputs of the ICERM workshop on Reproducibility on Computational and Experimental Mathematics [10], as well as some of the publications of the participants.


[1] One of the reasons for this is that up until very recently, it was difficult to conduct such a study - it was difficult to link software to a publication, few authors were publishing code, and it was difficult to data mine journals to assess impact. This is changing, an I expect studies to start emerging based on tools like ContentMine and ScienceToolbox.

[2] Piwowar, Day, Fridsma (2007). Sharing Detailed Research Data Is Associated with Increased Citation Rate. PLoS ONE.

[3] Gleditsch NP, Metelits C, Strand H. 2003. Posting your data: will you be scooped or will you be famous? International Studies Perspectives 4(1):89-97

[4] Pienta AM, Alter GC, Lyle JA. 2010. The enduring value of social science research: the use and reuse of primary research data. The Organisation, Economics and Policy of Scientific Research Workshop

[5] Henneken EA, Accomazzi A. 2011. Linking to data - effect on citation rates in astronomy.

[6] Dorch B. 2012. On the Citation Advantage of linking to data. hprints.

[7] Piwowar H, Vision T. 2013. Data reuse and the open data citation advantage. PeerJ. PubMed 24109559

[8] Howison J, Herbsleb, J. 2013. Incentives and Integration In Scientific Software Production. CSCW 2013.

[9] Howison J, Bullard J. How is software visible in the scientific literature? Preprint available from https://github.com/jameshowison/softcite/blob/master/paper/HowisonBullard-SoftwareCitation-WorkingPaper.pdf?raw=true

[10] "Setting the Default to Reproducible: Reproducibility in Computational and Experimental Mathematics," ICERM Workshop report, with D. Bailey, J. Borwein, R. LeVeque, W. Rider, and W. Stein.

  • 1
    Thanks for the detailed answer, Neil. I might add Donoho's "How to be a highly cited author in the mathematical sciences" to your references: in-cites.com/scientists/DrDavidDonoho.html He makes a fairly specific reference to publishing open source software as a means of improving impact. – Aron Ahmadia Feb 6 '15 at 18:28
  • " Whilst there were very few examples of people getting scooped".... I can't help wondering what examples those might be. – Faheem Mitha Feb 8 '15 at 12:54
  • Thanks Aron - I recall him having said something on that in one of his talks but did not know of that specific reference. And Faheem, there were two examples that I could corroborate - one was in the area of optimising communications in distributed computing, the other was instrument control in neuroscience (in this case the use of the software was acknowledged by the other group, but the author was beaten to a major research paper) – Neil Chue Hong Feb 9 '15 at 14:30
  • Nice answer and interesting references (+1). – Aleksandr Blekh Mar 19 '15 at 6:02

The Journal of Statistical Software is one of the few journals that publishes software. Currently, it has one of the highest impact factors among all statistics journals. This can be viewed as evidence that publishing software alongside research papers leads to higher impact and in particular to more people using the methods described in the paper.

  • "Currently, it has one of the highest impact factors among all statistics journals." xkcd.com/285 :-) – Faheem Mitha Feb 8 '15 at 13:03
  • 2
    I maintain a list of journals which publish software: software.ac.uk/resources/guides/… - in the three years I've been doing this the number of journals has gone from a handful to over 70. – Neil Chue Hong Feb 9 '15 at 14:31
  • To be accurate, JSS doesn't publish software per se, just relevant research papers. The same applies to @NeilChueHong's excellent list. But I get you both :-). +1 for mentioning JSS, which I refer to quite often. – Aleksandr Blekh Mar 19 '15 at 6:01

Your question can be interpreted in several different ways, and I'll try to address all of them.

  1. Is there evidence that by publishing software alongside research papers means that more people use the methods described in the paper?

This is incredibly apparent in bioinformatics software. My google scholar skills are not helping me at the moment, so I can't find a full academic paper on this, but based on personal experience and networking in both my narrow (RNA-seq) and broad (genomics) field, if a study describes an algorithm it will be cited and used only by people who are developing algorithms to solve the same/similar problem. If a study provides usable software: standalone, python package, jar, R library - something, it is much, much more likely to be cited and used (especially if it (a) works and (b) doesn't have any ridiculous dependencies (phyloCSF, I'm looking at you)).

I don't have time at the moment to do this analysis myself, but an easy pubmed scraping + text mining across several issues of the journal "Bioinformatics" should enable quantitative validation of this statement.

  1. The main argument in favour of this idea is that research should be reproducible, which has been addressed in other questions on this site

For many applied bioinformatics papers, including my own, we use other people's tools to do analyses, and write up the results. The code we usually use is "hacky" and not really "software", but is instead a script we ran that got us our results using our particular data and filesystem/server configuration. This code is - unfortunately - usually not published, and at the moment you are expected to write the detailed summary of what you did in the methods section of your paper. Unfortunately, all too frequently the people who are writing the paper (i.e. 1st author) is not the bioinformatician, and if the analyses were a small part of the work, the (usually biologist) 1st author tries to write up/summarize what the bioinformatician did based on his/her limited understanding (and the bioinformatician is not vested enough in the paper to care because he/she is 10th author) -- and if you don't get a bioinformatics-savvy reviewer the paper is published with the goobledygook in the methods, and no way to reproduce the analysis. I'd like to have a dollar for every time I've tried to understand/reproduce someone's methods, and not been able to because they were in a "Biology" journal and hence not described properly.

Going forward, I think more and more journals will start asking for code - but with the caveat that it won't be code that you can download and run - on their data or on yours (the data is too big, and writing code that would work on all possible clusters is too much effort) - so it will be more as a supplementary thing from which I can elucidate your exact methods as opposed to rerun your complete analysis.

3. This includes making the software used to perform research open source. Some of the best aligners for sequencing data are non open-source. A pubmed search for novoalign (proprietary) results in 15 hits, 1200 google scholar records. BWA (open source) 142 pubmed, 9960 google scholar. These results speak for themselves.

  • 1
    Detailed summaries are usually useless, even if written by people who understand what is going on, because they can't possibly include all the details required to reimplement the code. Plus having to reimplement someone's elses code for the purposes of reproduction falls into the category of cruel and unusual punishment. – Faheem Mitha Feb 7 '15 at 18:58

If a sample size of 1 counts as evidence, then yes, having open source software certainly has the potential for significant impact.

However, there are some important points to note before that potential for impact becomes real impact. Firstly proper software is not typical academic code thrown together with few comments to solve a specific problem. Software should have a user interface, user manual, be reliable and provide significant functionality. In my case I included several simulation methods within a single piece of software and wrote a review article detailing their implementation with examples. Since being open sourced last year many researchers have started using the software which will likely lead to citations of the paper. If you can develop a widely used software package then any methods you implement are (slightly) more likely to be used over ones which you do not. However if someone develops a significantly better method then not implementing that feature in your software may lead to people using competing packages if they exist.

I would say that if you want your method to be used or implemented, then including a sample code provides a much lower barrier to entry, for example popular random number generators always included source code. However, any code you write should be clearly licensed, and ideally as permissible as possible, eg BSD, so that it can be freely used in commercial and non-commercial software.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.