Popular proprietary program or obscure open source substitute for reproducible research?

Question

For a long time, I had been using open source software for my work to make it reproducible. I believed that if I made my code open source and the software to run the code was also open source (or at least free), my research would be reproducible. However, recently, in a discussion, it came through that research is more reproducible if one uses "popular" software instead of "unpopular" free ones.

For instance:

I had been using Scilab (free) for a lot of my work and distributed my files to others. But I was surprised that more people had MATLAB ($$) and preferred if I sent them MATLAB files instead (little modifications).

My question is :

Assuming I'm starting a new project and I wish to make it as reproducible as possible. Should I be using relatively unpopular free software or extremely popular proprietary ones?

Answer 1

I think there are two kinds of reproducibility:

1. The ability of someone else to run your code and obtain the same output.
2. The ability of someone else to write their own code that does the same thing as yours based on your description and on examination of your code (reproduction from scratch).

The second kind of reproducibility is much more convincing, since the main point of scientific reproducibility is to verify correctness of the result. For science that relies on code, it is usually impossible to include every detail of the code in the paper, so verification requires examination of the code.

If you use proprietary software, your code probably makes use of closed source code, and therefore it cannot be verified or reproduced from scratch. If you use open source software, then all of the code that your code calls is probably open source, so it can all be verified or reproduced by someone else from scratch.

At present, it is probably true that the first kind of reproducibility is more achievable with proprietary, widely-used software. I am optimistic that the current trend will lead to open-source software catching up in terms of wide use (consider SAGE, for example).

---

Addendum, in light of Epigrad's answer below, which I mainly agree with: The problem with relying on closed-source code isn't that someone else won't know what that closed-source code is expected to do.

The problem is that if you have two closed-source implementations of the same algorithm and they give different results (trust me, they usually will), then you have no way of determining which (if either) is correct.

In other words, closed-source code would be fine for reproducibility if it were bug-free. But it's not.

Answer 2

To supplement @David Ketcheson's answer with a "Yes, but..."

I agree that there are two types of reproducibility - CrossValidated discusses them with some degree of frequency. There is, as has been mentioned, "Can I click 'Run' and get the same answer you did" reproducibility, which I generally don't find very compelling.

There's also "Could I repeat your analysis from what you have provided from Step 1 to Step End, and get the same or a similar answer?" I think this is the one we should be aiming for.

That is often helped by using accessible, non-proprietary code, but not always. Consider the following example of an infectious disease dynamics model, expressed as a system of ODEs:

Here, in order to replicate (or fail to replicate) my findings, the software I used doesn't matter. What matters is the equations and parameter values I chose. If I provide those, then the only reason for code being needed is because someone doesn't want to implement the study from scratch, and does want to just run the code and see if the results match, tinker with the assumptions a bit, etc. In that case, everyone benefits from the code being in a form people use.

I think the same is often true for statistical analysis that doesn't use novel methods. At this point, what matters is that the data is available, and that the code is implemented in a language people understand and use. If 95% of people use SAS, even if it is proprietary, then the way to make your results most accessible, and most easy to replicate, is to have an implementation in SAS. Because if you pick an obscure but free language, what you've done is replaced the "Money" barrier with a "Time to understand" barrier - which for most people equates to the same thing.

The summary is this: I don't think "Free/Open" vs. "Proprietary/Closed" is necessarily the deciding distinction. I think that distinction is accessibility, and trying to maximize that. If there is both an open, free and popular software package that's used (R for example) then great! - use that. But if the field uses primarily one commercial package, picking an obscure alternative just because its free doesn't fix accessibility, it just shifts the burden.

Answer 3

Let me start with a disclaimer. I generally subscribe to the free software community perspective that proprietary software is questionable ethically, and best avoided if possible. I realise this perspective is not commonly held in scientific circles. Having said that, sometimes proprietary software is a necessary, or at least not easily avoided evil, and I'm generally pragmatic about using proprietary software when no good alternatives exist. I've used proprietary software in the past, though currently the only proprietary software I'm currently (sporadically) using is Skype, for which no good free alternatives exist.

However, special considerations apply in a scientific context. One of thse has already been covered by @David, namely that in general you can't "see inside" proprietary software to see how something is implemented. Having said that, sometimes proprietary software is written in an interpreted language, as in Splus, and one may be able to see part or all of an algorithm implementation. Regardless, the point holds generally.

A separate and obvious issue, which I don't think anyone has raised, is that using proprietary software forces others who want to use your software to buy the proprietary product you use. These products can be quite expensive, especially for people from poor countries. For example, Matlab, which has been mentioned in this thread, runs to thousands of dollars if one has to pay for a license oneself. Western academic institutions often have site licenses for such popular software, so researchers don't have to pay for it themselves. I personally am quite unhappy when I am expected to use a piece of software written using some proprietary language or package that has to be purchased.

A related issue is that much, if not most research, is done using public funding, i.e. taxpayer money. It seems undesirable to me to use such funds to buy proprietary software, thus adding to the profit of some corporation. In general, there is some movement to make academic work that is done using public funding free. And one can easily make the argument that the usage of proprietary software makes ones scientific product less free. For example, I believe the NIH now has some such policies in place. Similar arguments could be applied to the usage of software tools.

A tangential technical issue is that it is often difficult to get proprietary software to play nice on free software platforms such as the free Unix-like systems currently popular in scientific circles, e.g. the Linux based systems, and the BSD systems. These difficulties include, but are not restricted to

a) ABI problems. If one wants to compile a C/C++ extension for Matlab, for example, one has to use exactly the version of the compiler that the Matlab program has been compiled with

b) The program requires obsolete libraries or requires libraries to be in non-standard places.

I mention this issue in part because my understanding of the question is that it is asking about proprietary vs free in the context of pragmatic usage.

So, to respond to the question directly:

Assuming I'm starting a new project and I wish to make it as reproducible as possible. Should I be using relatively unpopular free software or extremely popular proprietary ones?

I don't think there is a clear answer. If there is no viable alternative, then one would have to use the proprietary software, as I do with Skype. If there a viable free version, I would use it. Bear in mind that if more people start using the "relatively unpopular free software" it will become more popular. :-)

Answer 4

Agreeing with most that has been said by EnergyNumbers and David Ketcheson, I'd like to add some slightly different points:

• the fact that code is written in a particular language (Matlab) does not make it closed source per se.
  Just as using Scilab on closed source Windows (or using a closed source BLAS) doesn't make Scilab closed source.
  There are journals that require reproducible research and open code and accept Matlab code.

• likewise, popular and free do not exclude each other, nor does proprietary imply that it is popular

• neither of the two imply that the respective software is suited for the reproducible data analysis.

• The choice on the language should be based on several factors

  • how suitable is it for the problem at hand
  • here also: how suitable is it for reproducible data analysis (I'm experimental scientist, so reproducing a data analysis is just a part of reproducible research for me)
  • particularly if talking about sharing code: infrastructure considerations (can code be packaged into libraries? How can data and code and the text be bundled into a reproducible paper?)
  • popularity = size of peer group using this software (which somehow includes the cost of a license)
  • you may also want to consider what the peer group that is interested in reproducible research is using (In my field(s), the reproducible research crowd coincides much more with the open source (R) crowd, while of the peer group working on the same kind of problem probably the majority uses Matlab)

• My old supervisor used to say that the cost of a license is no scientific argument.
  But of course you may need to consider it.

• Likewise, popularity is not a scientific argument, but you should critically examine whether the propularity does actually indicate that lots of good reasons exist to use that software.

Examples:

• In my field, R is increasingly popular (by now popular enough to publish in R) and to a certain extent replacing Matlab.

  • So R is both free and popular (yet most people from my field don't make use of the fact that you can look into R's source if they discuss reproducible data analysis)

• Reasons IMHO include

  • most importantly: R is well suited for our kind of problems (I think it is better suited than Matlab, others differ slightly in their opinion and use Matlab. Even others do think it may actually be better suited but not as much as to outweigh learning R right now)
  • Being well suited includes the availability of methods we use at CRAN vs. commercial Matlab toolboxes and Matlab file repository
  • Being better suited includes the fact that I cannot adapt the code of proprietary Matlab toolboxes (p files) to particular needs.
  • Being well suited includes the ease of report generation
  • infrastructure: e.g. R's package concept enforcing a minimal standard and allowing to rely on examples and tests actually being runable vs. a folder full of .m files (I heard that recently a more package like concept was introduced). This helps a lot with sharing the code (whether to reproduce your findings or to use it on their own data)
  • license costs come in indirectly: it is more that you don't need to worry if you install it on your private computer as well and you can tell students to install it on their laptops than worrying about the cost of a few licenses for computers at work.
  • probably far more costly than license fees themselves are the time to get the license manager working or to transfer licenses between computers, and if you just want to give a toolbox a try before deciding whether to buy it or not, the hassle of a) asking the vendor for a demo version and later on b) of filling out order forms and writing justifications why you need to spend money on that license.

Note how many of these arguments are "soft" and in fact have more to do with being used to one system or the other or using a feature is easier, more obvious, or more common among the peer group of that software's users: noweb can work with Matlab, documentation of m-files is possible and encouraged (though not really enforced), unit tests are possible in Matlab, Matlab Central has lots of free code, etc. Just R users always know of CRAN, whereas not all Matlab users know of Matlab Central, there's a good culture of citing scientific papers about the implemented method in R help files, shipping code together with example data sets and/or giving actually running example code.

Examples of invalid arguments:

• if my peers did not use version control for coding, I wouldn't consider that an argument against using version control (as there are lots of good reasons for using it)

• Or, if the ones refusing to use version control were programming in Matlab, neither would that be an argument against Matlab - but I'd check whether there is any reason that prohibits using version control for Matlab code.

Answer 5

Listen to your colleagues & peers.

They've already told you what is most suitable for them, in order for them to be able to reproduce your results.

That answer, in your particular case, is Matlab.

There will be some others who want to port it to Octave, SciLab, Excel, Fortran or whatever. That's fine too - but if you're flexible about which platform you code in, and coding in Matlab won't make you less productive (or the small reduction in productivity is a price worth paying for the increased reproducibility), then code in Matlab. Because your colleagues have already told you that that's what enables them to reproduce your work, easiest.

There are plenty of good reasons (and maybe some bad ones) why many of your colleagues prefer Matlab. Sometimes the cheapest things can cost you most.

For anyone else reading this, with a similar problem, the gist of the answer is the same: listen to your peers.

Answer 6

My two cents:

I often compare code with a scientific paper. The purpose of a paper is to describe your results and approach to the problem in such a way that your peers can validate/refute your findings, in whatever way they see fit, so that a collaborative effort can take place to make progress in the field.

Who cares whether the paper's author has used LaTeX to write his paper, or MS  Word? Who cares if the data was processed with MATLAB, or Excel, or Pascal? It is the truth(s) in the paper that count(s), not the tools used to get there (although many would gladly jump in here and start a fierce discussion on this point...but in my experience, the arguments used in such discussions tend to be more like religion than anything else).

What is very important however, is the means of getting through to your peers. For example, if you write and publish a paper in Esperanto (supposing for the moment that would get accepted), simply because you think it's beautiful and elegant and everyone should speak it. Plus there are many books about how to convey meaning in Esperanto right?

Not many of your peers will be able to understand the paper, let alone get the message and reproduce your findings. You'll have to wait until someone comes along in your field who shares your views on Esperanto, which might take half a lifetime. Altogether this is a very poor way of making progress in your field.

This I think is the crux of your predicament -- if your peers mostly use MATLAB, you'd best stick to MATLAB, because you'll reach the most peers the quickest. Leave it to (other) engineers to sort out whether MATLAB actually produces good (enough) results for your case, and leave it to one of your peers half a world away to translate your code to C++ and verify your findings.

It's not the code that counts -- it's the knowledge that is contained in it.

Answer 7

You can do whatever you damn well please, but there are a few considerations:

1) You might have an obligation to disseminate your work. If you are supported by an external funding agency such as the NSF or NIH then dissemination is an obligation. Many private foundations and other funding sources also provide support with the intent of dissemination, whether it's stated explicitly or not.

If you have such an obligation, then absent any other consideration this would suggest you use the most accessible software possible, regardless of whether that is proprietary or free.

2) Open source-ness is important for some kinds of research, but for most research it's irrelevant. Unless you're doing computer systems research, where it really matters how the computer arrives at an answer, the method does not matter so much as correctness. Sometimes it can matter (e.g. if your work is heavily dependent on numerical methods), but it probably doesn't.

3) Communities establish standards of validity and integrity. If everyone else uses MATLAB, then the community has deemed it accurate. Absent evidence to the contrary, using open-source software does not make your results seem more correct or more verifiable in anyone's eyes.

As as side note, Mathworks has a strong reputation for working with researchers. If you really felt that MATLAB was giving you incorrect results, and you had examples to show it, they would be knocking down your door to fix the issue. I've had Mathworks issue me a custom support patch the very same day I called about an obscure hardware incompatibility that was causing incorrect behavior.