Are virtual machines good for reproducibility and open science?

Question

Virtual machines (VMs) provide a way for scientists to package not only scientific software, but also data, external dependences, and even entire operating system configurations, facilitating a faithful and exact reproduction of a particular computing environment used to derive a particular result.

Are VMs and other container systems used in this way actually a net positive for open science? If the scientific software used to compute a result can only be replicated in a very particular computing environment, is it useful and reliable?

Answer 1

I like to think of VMs, containers, and the like as allowing repeatable research. I don't consider it reproducible if the results can only be repeated by having exactly the specified VM/container used by the original researcher.

That doesn't mean VMs or containers are not useful for open science; they do provide a means for people to rapidly disseminate the computing environment within which the results were produced. They also provide a point of reference should software used change at a later date which causes repeatability issues. If open source software is used it would be possible to track down what changed and why this affected the particular results.

At the very least, VMs and containers provide a good means with which to document the computing environment used to generate the published results.

Answer 2

Well, Yes, but I would say that wouldn't I, since I wrote the Recomputation Manifesto. about this.

While the recomputation manifesto focussed on reproducibility, I think if anything it's more important for Open Science than for raw reproducibility. Because you can see inside the experiments behind some claim, and see if they are valid experiments or not. Also - and this is a point not emphasised enough - making available experiments and code etc behind a paper in a VM (i.e. in working form) enables people to build on them and stand on the shoulders of giants.

Reproducibility by VM has been criticised as a very limited form of reproducibility, see e.g. Titus Brown's blog post. That is indeed true, but I would also respond: "Sure, but if we can't even bother to reproduce the exact experiment done, what hope is there for richer reproducibility?"

Also I would argue that even this relatively poor form of reproducibility could be critically important. For example, if we could do the physical equivalent of recomputing the exact original Cold Fusion experiments, we might know what happened to cause those results.

Answer 3

CERN have been running such VM-based reproducibility infrastructure for a decade (overview) and found it useful enough that they recently scaled it up in the cloud, so that may be a good place to start digging deeper.

In the biomedical field, systems like Docker have been found useful and reliable enough that they got some traction.

Given the complexity of preserving executability in the long run, I think the focus should be on ensuring that relevant code has been independently run in a somewhat contemporary manner, with results documented in a way that is suitable for long-term preservation. This is the approach chosen at places like Research Compendia, and getting this to work at scale would seem to me like a great net positive for open science.

Answer 4

Virtual machines are good for dissemination, and irrelevant for reproducibility.

What virtual machines facilitate is running the original researcher's software and dataset.

In terms of open science, virtual machines make it easier for others to understand the original result and derive other results. To derive other results, the full methodology used to derive the software package needs to be made available, including software sources and configuartion parameters. Virtual machines are useful in this respect because they provide a reference configuration: often the exact sequences of steps to install the system are not tracked as the system undergoes maintenance, but configuration files can be extracted from the VM image.

For reproducibility, rebuilding the original researcher's software in a different environment is not enough. There is no assurance that the original software implements what the research article claims. Reproducing the result entails writing independent software to verify the claims. Having a virtual machine available doesn't hurt the ability or desirability of writing independent software.

Answer 5

If nothing else I find building a Docker image of your project a good way to track all your dependencies. Otherwise, it is easy to forget e.g about that little R function that assumes you have a particular unix tool like "imagemagick" installed on your system, or forgetting to include your latex bibliography... here's an example: https://hub.docker.com/r/sje30/waverepo/