How should I reference a repository of data and resources created to accompany my publication?

Question

I'm writing a research paper and there's a large amount of code and data related to the study which may be of use to those who would read the study. They can confirm the correctness of the techniques, modify it for their own purposes, perform new analysis on the data, etc.

All these relevant files exist in a public repo, but I'm uncertain what's the best way to reference these. Would it be a citation? Just drop the URL directly into the paper? Other ideas?

Answer 1

Short Recommendation: Include the URL in-text followed by a full citation and end of text reference. If using someone else's repository, do the same, and probably also include a citation to the accompanying paper, unless the authors request otherwise. Place this citation in a prominent position in the method. If the open repository is a big part of the contribution of the paper, consider also including the URL in the abstract.

Longer Answer

Researchers in my field of psychology are starting to use the Open Science Framework as a repository for archiving data, code, and other materials. These repositories include a short URL that is meant to be stable (e.g., https://osf.io/5krfq/ ).

When referencing these repositories, I have read papers that just include the URL, and others that include a full citation. For example, the full citation for the above using APA style might be:

Wynton, S. K. A., & Anglim, J. (2017). Abrupt Strategy Change Underlies Gradual Performance Change: Bayesian Hierarchical Models of Component and Aggregate Strategy Use. Retrieved from https://osf.io/5krfq

So in summary, I don't think the standards for data and repository citation have been formalised sufficiently yet.

I think there are a few considerations:

• If there are any questions about the stability of the repository URL, then you need to provide more information. For example, if the repository is hosted on a standard university website, then it is quite likely that the URL might change over time. Even in the case of something like the OSF, it might still be safer to include additional information. References work well because there is redundancy.
• In general, providing a URL in the main text makes it easier for the reader to see that the repository is readily accessible.
• In many performance evaluation systems, citations of papers are counted, whereas citations of other resources may not be. This may be changing, but it is worth thinking about.

Referencing existing repositories by other authors

If you are referencing an existing repository that other authors created, then you should look to see what requests these authors made. For example, they may want you to cite a particular paper that is linked to the repository. More generally, there is a arguably an ethical and professional obligation to acknowledge those providing the repository by using a full citation.

Referencing your own repository that accompanies a paper

If you are creating your own repository that is linked to the paper you are writing, I quite like the idea of including the URL in text as well as the end of text reference. E.g.,

Data and code is available at https://osf.io/5krfq (Wynton & Anglim, 2017).

And then you include the end of text reference as above.

This has the benefit of making the URL very clear to the reader (e.g., encourages the reader to click the link) but it also has the benefits of full citations (e.g., the reference is more robust, creates a practice of citing data and code as equal to citing papers).

Note about blind review: If you are submitting your manuscript to a place that does blind review, you need to do another step during the review process to prevent disclosure of author identities. One approach is to put a black mark over the author names in the in-text citation and end-of-text reference. OSF also has the benefit that you can create a custom-URL that provides a read-only view of the repository with author names removed. When such a feature is not provided by your repository service, you might need to instead attach anonymised versions of the materials and black out the link.

Where to put this citation/reference? There is another issue of where to put this reference. E.g., Abstract, author note, first sentence of the method, some other section of the method, etc. There are several considerations:

• More people will see it if you put it in the abstract. The author note is another relatively prominent location. Thus, if you see the repository as fundamental to the contribution of your paper, then you may want to include the url in your abstract, and then include the full citation somewhere in the method.
• Some journals have conventions regarding this. e.g., some journals have badges for open data or open materials, some journals ask for a section in the method with a label like "Open Practices". If so, then it makes sense to follow these conventions.
• If you want to draw some attention to your open data and materials (but putting the url in the abstract feels excessive), then I think the first sentence of the method possibly under a section heading like "Open Practices" is a good option. Thus, it will be clear to any reader that gets to the method that these materials are available.
• If you think the repository is not that important, then it could be placed in whatever section seems most content relevant (e.g., end of the participants/data description section or somewhere in the section discussing the data analytic approach).

Personally, I like the idea of "Open Practices" (or something similar) becoming a standardized section in manuscripts where the authors explain what materials have been made open or are made to justify why they are not open.

What name to give to repository reference: Another issue, which I have not yet resolved, is what is the best title for repositories that accompany a paper. Possible titles:

• Identical title as accompanying paper
• Related title to accompanying paper: e.g., "Data and code for [insert paper title here]" or "Supplementary materials for [insert paper here]"
• Something very descriptive: e.g., "Data and code examining ..."

If you use the identical title as the accompanying paper, this creates the potential for ambiguity when people cite the paper or the repository. However, it might make it easier for people to find. And if getting citations to your original paper rather than the repository is particularly important, then it may be the case that citations to the repository will get counted towards the paper depending on how the citation engine (e.g., Google Scholar, Scopus, etc.) matches articles. Such repositories can also often host a pre-print of the article. In that sense the repository becomes almost a landing page for the actual publication that is not behind a publisher's paywall.

From a descriptive perspective, I think that something like "Data and code for [insert title]" seems better. It makes the link with the accompanying paper very clear, but also makes it clear that it is a distinct academic artefact.

Answer 2

I typically have a subsection at the end of my methods section that is like this:

Reproducibility and open source materials

To enable re-use of our materials and improve reproducibility and transparency according to the principles outlined in Marwick (2016), we include the entire R code used for all the analysis and visualizations contained in this paper in our SOM at http://dx.doi.org/10.17605/OSF.IO/RTZTH. Also in this version-controlled compendium are the raw data for all the tests reported here, as well as additional regression diagnostics and power tests. All of the figures, tables and statistical test results presented here can be independently reproduced with the code and data in this repository. In our SOM our code is released under the MIT licence, our data as CC-0, and our figures as CC-BY, to enable maximum re-use (for more details, see Marwick 2016).

That specific paragraph can be found in this published paper

Marwick, B., et al. 2017. Movement of lithics by trampling: An experiment in the Madjedbebe sediments, northern Australia. Journal of Archaeological Science 79:73-85. http://dx.doi.org/10.1016/j.jas.2017.01.008 preprint: https://osf.io/preprints/socarxiv/7a6h6/

And minor variants of that paragraph can be found in most of my recent papers. By including this kind of paragraph in my publications, I am trying to fulfill the recommendations in this paper, which echoes ideas found in many other similar manifesto-like papers on reproducibility:

Stodden, Victoria, et al. 2016 Enhancing reproducibility for computational methods. Science 354(6317):1240 http://science.sciencemag.org/content/354/6317/1240

The Marwick 2016 citation in the paragraph above is my in-depth discussion of computational reproducibility for archaeology:

Marwick, B. (2016). Computational reproducibility in archaeological research: Basic principles and a case study of their implementation. Journal of Archaeological Method and Theory, doi: 10.1007/s10816-015-9272-9, preprint: https://osf.io/preprints/socarxiv/q4v73

Answer 3

If the rest of your paper is using MLA/APA/Chicago/etc. citation, then these sources should be treated no different. You should maintain consistency throughout your paper.