1. Reproducibility is enhanced by following best current practices, including:

 

a. Ideally, exclusive use of publicly available data. However if the research domain does not allow for publically available data for widely accepted reasons (e.g., medical data with high confidentiality concerns), the principles outlined in items (b) - (e) should still be followed;

 

b. Use of version control for all (collaborative or individual) code development;

 

c. Exclusive use of open-source software freely available to anyone in the world

 

d. End-to-end scripting of research, including data processing and cleaning, statistical analyses, visualizations, and report and/or manuscript generation, with the full workflow made available to others;

 

e. Use of container/virtual machine tools to capture software versions, dependencies, and platform specifics;

 

f. Publication of code in public repositories as with data; and

 

g. For projects that develop algorithms, implementing algorithms on standard computational platforms (e.g., R packages, Python packages, source code packages installable via standard methods, etc.).

 

2. Reproducibility shouldn't be thought of as a binary state: either reproducible or not-reproducible. It's more useful to think about it as continuum from hard to reproduce to easy to reproduce. The goal of any reproducibility effort should broadly be to move as many people as possible further towards easy-to-reproduce. This involves some technological (to make the right thing easier than the wrong thing) and some social (to give people the activation energy to learn a better process even though it's harder in the short-term) components.

3. It’s perhaps worth noting that in this era of “replication crisis”, reproducibility is the only thing that can be effectively guaranteed in a published study. Whether any claimed findings are indeed true or false can only be confirmed via additional studies, but reproducibility can be confirmed immediately.

1. Data anonymisation/de-identification: Data holders are responsible for generating de-identified datasets which are intended to offer increased protection for patient privacy through masking or generalisation of direct and some indirect identifiers.

2. Controlled access to data, including use of a data sharing agreement: A legally binding data sharing agreement should be in place, including agreements not to download or further share data and not to attempt to seek to identify patients. Appropriate levels of security should be used for transferring data or providing access; one solution is use of a secure ‘locked box’ system which provides additional safeguards.

1. Reproducibility is enhanced by following best current practices, including:

a. Ideally, exclusive use of publicly available data. However if the research domain does not allow for publically available data for widely accepted reasons (e.g., medical data with high confidentiality concerns), the principles outlined in items (b) - (e) should still be followed;

b. Use of version control for all (collaborative or individual) code development;

c. Exclusive use of open-source software freely available to anyone in the world

d. End-to-end scripting of research, including data processing and cleaning, statistical analyses, visualizations, and report and/or manuscript generation, with the full workflow made available to others;

e. Use of container/virtual machine tools to capture software versions, dependencies, and platform specifics;

f. Publication of code in public repositories as with data; and

g. For projects that develop algorithms, implementing algorithms on standard computational platforms (e.g., R packages, Python packages, source code packages installable via standard methods, etc.).

2. Reproducibility shouldn't be thought of as a binary state: either reproducible or not-reproducible. It's more useful to think about it as continuum from hard to reproduce to easy to reproduce. The goal of any reproducibility effort should broadly be to move as many people as possible further towards easy-to-reproduce. This involves some technological (to make the right thing easier than the wrong thing) and some social (to give people the activation energy to learn a better process even though it's harder in the short-term) components.

3. It’s perhaps worth noting that in this era of “replication crisis”, reproducibility is the only thing that can be effectively guaranteed in a published study. Whether any claimed findings are indeed true or false can only be confirmed via additional studies, but reproducibility can be confirmed immediately.

1. Data anonymisation/de-identification: Data holders are responsible for generating de-identified datasets which are intended to offer increased protection for patient privacy through masking or generalisation of direct and some indirect identifiers.

2. Controlled access to data, including use of a data sharing agreement: A legally binding data sharing agreement should be in place, including agreements not to download or further share data and not to attempt to seek to identify patients. Appropriate levels of security should be used for transferring data or providing access; one solution is use of a secure ‘locked box’ system which provides additional safeguards.