How can I show which data I'm using in my paper?

Question

I've just started reading papers about speech recognition and algorithms on medium sized graphs (~800,000 nodes and 4,400,000 edges with some connected text data). I think a problem of these papers is quite often, that it is difficult to check the experimental results. It is difficult to check them because of two reasons:

1. The source code that was used to generate the results is not (publicly) available
2. The data is either not at all available or it is not clear which version of the data was used

When I start writing papers, I would like to make it easier to check the results.

The first problem is easy to solve: I can simply provide the source code (e.g. on GitHub or my personal web space).

The second problem seems to be much harder to solve. The data is often quite big (speech recognition: several GB; graphs: about 2GB). This is too much to upload it on my personal webspace / GitHub.

How can I show which data I'm using in my paper? (Currently I give a link to the data source and note the data when I've downloaded it. Additionally, I note the date/version of the source if possible.)

Are there projects that try to solve this issue? (e.g. by providing space for important / interesting projects like dblp, a version history and good download speeds)

Answer 1

Both the problems you state are significant, and (I think) very problematic to the further progression of computer science as an actual science. I work in services / software engineering, and quite honestly, reproducibility of results is very, very bad in my work as well as in the work of others.

I will comment on both of your issues, starting with the second one:

The data is either not at all available or it is not clear which version of the data was used

In many cases, this is actually the easier-to-solve issue. I would assume your department has IT resources (e.g., a department web space) that you can use for such purposes. Additionally, there are other, more field-specific, repositories that you can upload data to (for instance, there is the UCI machine learning data set repository or the Grid workload traces archive). Making data available to the public is really only an issue if you are not allowed to do so because of commercial interests of your data provider. For instance, I have access to real traces of executions of business processes of a big german logistics provider, which I am not allowed to give out. This clearly limits the usefulness of these traces to me.

The second problem

The source code that was used to generate the results is not (publicly) available

is sort of tricky in practice. Clearly, throwing your code to Github is easy, but this hardly solves the issue of reproducibility. You will still need to include pretty detailed usage instructions and documentation to make this code of any use to another researcher. This may seem trivial for a naive 1000 Lines-of-Code implementation of an algorithm, but, for instance, my current research prototype approaches 25.000 lines of Java code plus a bit of XML and Groovy. Just putting this up on a repository is not enough, if we consider that a paper can only be considered reproducible if it is possible to do so in reasonable time. An additional problem in my field is that often prototypes are built for a specific execution environment. For instance, my department has a small OpenStack based private cloud, and many demonstrators that our students build realistically only run in this environment without drastic modifications.

I am currently in the transition between a postdoc and a more independent research position, and one of my personal goals is to make all work that I and my students publish easier to reproduce. However, so far we are only making baby steps, but at least we are trying :D

Answer 2

Recently (at least) two platforms have appeared which offer web space for sharing scientific data. They provide Digital Object Identifiers (DOI) for uploaded datasets which can be used to refer to the data easily.

• Figshare seems to be the more prominent option. All data is shared under CC0 license. There is a 1 GB limit for private data and no limit for public data.
• Zenodo is part of a European research project. You can specify any license you prefer for your dataset. Currently they have a 2 GB limit per file but you can upload as many files as necessary. For larger files, you need to contact the administrators.