Reproducible and exact calculations vs. approximate and slightly random results in experiment

Question

When experiments are performed and reported in an academic article, I learned that reproducibility is important. I am implementing a system where I stand between the choices to use GPU acceleration to perform certain calculations, where reproducibility can not be guaranteed, or to perform the calculations on the CPU, whereupon the calculations are exactly the same. The problem with using the CPU is that it is much much slower. However, it seems that the calculations on the GPU are about the same at each run.

I have the perception that researchers sometimes have to choose between qualitative or quantitative measurements. This may seem obvious to experienced researchers, but what I am asking is if a possible solution would be to perform a number of measurements on the GPU that provide estimates, and then calculate an average of the results? Is this "scientifically accepted", although calculations depend to some extent on randomness?

I should add that the results are only going to be part of a project at bachelor level, but I'm also interested in how it works in "real life".

Answer 1

It's probably best to answer a generalised form of your question first before getting to specifics.

---

Reproducibility is a tricky issue in applied Computer Science but the principles are the same.

The first thing you need to consider is what, precisely, the results of your experiment depends on:

• Does it depend on hardware? (e.g., performance experiments)
• Does it depend on data?
• Does it depend on software? (e.g., an implementation of an algorithm)
• Does it depend on configurations? (e.g., amount of memory set)
• ...

For whatever your experiment depends on, in an ideal case, reproducibility should make those materials available to third parties, either through links from the paper, or as details in the paper itself. However, hardware in particular is often a tricky issue in that you're probably not going to make the machine you ran the experiments on available. For this, it is typical to try use conventional hardware insofar as possible and to provide details in the paper.

And sometimes there are practical obstables for even sharing data or software, such as commerical interests, patents, licencing of the software used, and so forth.

---

In terms of reproducing your results, another aspect to consider is the stability of those results. When someone reruns the experiments, you would like them to see the same (or "negligibly different") results as you published. This means that, for example, if your results vary in each run, you should publish some bound or confidence interval on each run to give an idea of the distribution of results over each run, or you should take a measure like the average of multiple runs to ensure a stable enough result from which to draw stable conclusions.

---

However, what does "negligibly different" mean? Well while results should be "as reproducible as possible" (one will never get the exact same runtimes, for example, in a performance experiment), what is more important to the scientific method is that conclusions are reproducible. For example, if a bunch of doctors produce a study of 1000 patients showing a significant and strong correlation between eating popcorn and cancer, it is not necessary that the details of the study be reproducible but rather it is even preferable that the correlation be reproducible (and perhaps strengthened in a further study to look for causation by isolating the bad part of the popcorn and injecting it into some unfortunate mice or something).

So if you have general conclusions, then you might want to think about how reproducible they would be on different hardware, on different data, on different software, with different configurations, etc. If you don't know how the results would change when a particular variable changes, then you need to include that variable in your conclusion. That is to say, if you don't know how the result would change for different data, for example, then all you can conclude about is the data you've experimented with and maybe propose a stronger result.

---

Of course, almost all applied CS papers I've seen (including my own) don't stick to these rules because peer-reviewers don't hold them to such standards, other than in the most blatant of cases.

And it is exceptionally difficult to account for all variables in a thorough conclusion since in applied CS there's so much to take into account, particularly for performance: the machine(s), the programming language(s), the compiler(s), how good the programmer was, possible bugs, the data, other things running in the background, caching, ... hence reviewers tend to be a little more relaxed (perhaps sometimes too relaxed).

---

Summary: Publish both results if you think they are interesting. Publish as much material as you can to allow the results to be reproduced. If you're worried that the GPU results are too variable in each run to be reproduced, take the average of multiple runs to get a more stable result, or provide an estimated upper and lower bound, or something to characterise the variance.

But all of that is just the icing on the cake. The real goal of reproducibility is to be able to reproduce conclusions. Be careful to craft conclusions that do not overgeneralise the data they are drawn from.

Answer 2

I'm no computer scientist. But it seems to me that you have two methods,

Method A is fast but has some random error.

Method B is slower but more (totally?) accurate.

One approach, then, would be to determine what level of error is acceptable in your outputs, run a small experiment with both, and look at the differences between the results of each to find the level of error that is present in method A. If the level of error that you find is less than that which you decided was acceptable, then you can go with A for the speed.