How can we change things so more researchers are willing to share their source code?

Question

This is a followup to Why are CS researchers reluctant to share code and what techniques can I use to encourage sharing?. That question specifically asked how one can succeed in obtaining researcher's source code.

As discussed in the answers to that question, the reasons largely boil down to competitive advantage and people thinking their code is not good enough. The former issue is difficult to address. However, one could try to address the latter issue, making the reasonable assumption that this behavior stems from the surrounding academic culture. There may be additional aspects of the academic culture that discourage code sharing, and which do not relate to competitive advantage.

So, one could instead ask the general questions what concrete actions one can take to change this culture? Or, to put it a little differently, how can I help change the academic world so that more researchers are willing to share their source code?

Answer 1

In the long term, this will only happen if you change the culture (just as you say). How do you change the culture of a field? Very slowly, and only with enormous effort. You talk to other researchers. You articulate your values, and seek allies who share your values. You patiently make the case to your fellow colleagues, perhaps by writing opinion pieces. You don't harangue or attack them; instead, you gently lay out the reasons why it is good for science and good for them to share their code. Remember, in all likelihood you all share the same common values: the love and dedication of science and the pursuit of truth.

You lead, by acting as a model for how you would like others to behave. You do what you think is right, and set a good example.

You try to persuade referees and program committees to value and reward researchers who do share their source code. Recognize the amount of extra effort this takes, and (if you believe it is valuable) reward it accordingly: bump up your rating of their paper correspondingly, and make the case why others to do so. When you write letters of reference or evaluation for another researcher, if they share their source code, give them kudos and explain why the hiring or promotion committee should view this positively.

Ultimately, this is not something that a single individual can change. Only the entire community, acting as a whole, can make this kind of change. Individuals can catalyze and facilitate that change, but like any other kind of reform, it takes extraordinary patience and effort, as well as buy-in from your colleagues. It's not unusual for this kind of change to take a generation or two. But keep your chin up: remember, you're doing this because you believe it is good for science and good for your field!

Answer 2

IMO talk about "cultural" impediments are overstated. Academics are as rational agents as any, and currently the academic system (mainly publish/get grants or perish) provides little incentive to publishing the code or making analysis entirely reproducible. Some fields have started intiatives to encourage this by making either analysis and publicly releasing data mandatory or strongly recommended for publication (e.g. The Journal of Applied Economics) or for funding (e.g. NIJ grants frequently have stipulations to post the data to ICPSR).

Greater awareness of technical computing skills necessary for reproducible analysis will help (see Koeneker & Zeileis, 2009), but on its own won't spurn greater compliance, even if the already discussed negatives are largely mitigated. It still will be more work to publish the code than to not publish the code. When it helps your tenure case, then it will become more commonplace.

Answer 3

Researchers are unwilling to share their code because it's a lot of extra work -- for which there is little or no recompense. When I write some to simulate an experiment or an algorithm to verify numerically the result of a calculation, it is not production-ready code that can be easily run in another environment: at the very least another researcher needs Matlab or Mathematica, they need whatever special toolboxes or auxiliary code I am using, they need the data files in the right place on the hard drive, they need to understand how to program themselves so that when some small glitch arises, they can deal with it. When I try to run my own code from a year ago, it almost needs some massaging: perhaps a needed file has been moved from one directory to another, perhaps a toolbox has had its code base "updated" and no longer works exactly the same way.

So -- here's what usually happens. Someone contacts me and wants to try out my code. I warn them of all the above issues, but they insist that they know what they are doing and will get back to me with any problems. I spend 2 or 3 hours preparing things, checking stuff out so that they will have an easier time, explaining to them how to put things together so that it will all work. I mail it to them. And I never hear back. It happened again last month. So -- how likely am I to "share" code in the future? Just a little bit less likely than last month.

Now to the question: how can you get researchers to share their code? First, when you ask, follow through - don't "take the code and disappear." Second, try to get the researcher interested in why you want the code, what you might do with it. Third, the burden is on you to take research-style code (poorly commented, bad error checking, disorganized structure) and to make it do something. Fourth, return something: when you do make some progress, let the researcher know. Fifth, don't ask for impossible things: can I compile the code for your machine (that's different from mine)? (Hint: no).