What is the difference between repeatability, replicability and reproducibility?

Question

I have seen many instances where authors used the term "reproducibility" and "replicability" interchangeably in the social and behavioural sciences. Sometimes, they distinguish between the "repeatability" of experiments (same measurand/same measurement conditions) and "replicability" (same measurand/different conditions). If the three concepts are differentiated than, in most cases, there seems to be an inherent hierarchy between the three concepts:
repeatability < reproducibility < replicability Where a successful replication means that the same finding has been achieved with different data (or sometimes methods) and reproducibility means that it is possible to get the same results given the data and analytical means from the original study.

However, it occurs to me that at least in computer science this seems to be different.[1]
I am not aware of the situation in other fields. Therefore, I am curious which definitions of repeatability, reproducibility and replicability are used in other disciplines.

What are the definitions most commonly associated to repeatability, reproducibility and replicability in your field? Are the definitions the same but the concepts have substantially different meanings between fields e.g. because pseudo-random numbers generated by computer experiments are different from true randomness in biological experiments?

Answer 1

The American Statistical Association (ASA) has developed Reproducible Research Recommendations. The purpose of these recommendations are to create transparent science that can be recreated independently of the data creators and analyzers. These recommendations have been motivated by several high profiles examples of studies being either retracted or not repeatable. The ASA includes definitions for Reproducibility and Replicability:

Reproducibility: A study is reproducible if you can take the original data and the computer code used to analyze the data and reproduce all of the numerical findings from the study. This may initially sound like a trivial task but experience has shown that it’s not always easy to achieve this seemingly minimal standard.

Replicability: This is the act of repeating an entire study, independently of the original investigator without the use of original data (but generally using the same methods).

I do not know repeatability fits into these definitions or if it has a formal definition. Also, this hierarchy seems to differ from yours. Reproducibility is using exiting data and recreating the same results using the described methods. Replicability is conducting a new experiment and reaching the same conclusions.

On a personal note, I have tried to reproduce studies while working with a statistician and most life science journal articles do not include enough details to exactly recreate analysis unless they include their script (e.g., Python, SAS, or R code used to analyze the data). We were trying to recreate simple linear regressions and ANOVAs to find case studies for undergraduate stats courses, but often got different regression coefficients and test statistics. For example, people often make assumptions about NA values or transformations that are not described in their formal writeup.

For examples of replicability problems, search for psychology reproducibility problems. Also, notice the inconsistent use of the r terms here.

For examples of reproducibility problems, this lecture describes how a forensic statistician uncovered the Duke Cancer Scandal a few years back. Although outright fraud was occurring, there were other serious issues with the data analyzes.

Answer 2

In computaitonal research I would argue that something is "reproducible" if I can rerun your analysis with your code and get the same answer as you did. This might sound trivial, with differences really only being connected to pseudo-random number generation, but in fact this is far from the truth.

• Many computational studies do not provide their code.
• Even where they do, recreating the exact compilation/execution environment is next to impossible.
  • Are you using the same compiler and version
  • Are you using the same numeric libraries (BLAS/MKL) and the same versions
  • Does your system use the same precision in its floating point calculations
  • etc etc etc

You give me your code and enough information for me to produce and identical environment or (even better) your code is insenstive the environment, then your research is Repeatable.

If you describe your study sufficiently well that I can re-implement your study from scratch, without looking at your code and still get the same answer, then it is Reproducible

If I can arrive at the same conclusions as you, just from a description of its aims, then it is Replicable.