5

I've just started reading papers about speech recognition and algorithms on medium sized graphs (~800,000 nodes and 4,400,000 edges with some connected text data). I think a problem of these papers is quite often, that it is difficult to check the experimental results. It is difficult to check them because of two reasons:

  1. The source code that was used to generate the results is not (publicly) available
  2. The data is either not at all available or it is not clear which version of the data was used

When I start writing papers, I would like to make it easier to check the results.

The first problem is easy to solve: I can simply provide the source code (e.g. on GitHub or my personal web space).

The second problem seems to be much harder to solve. The data is often quite big (speech recognition: several GB; graphs: about 2GB). This is too much to upload it on my personal webspace / GitHub.

How can I show which data I'm using in my paper? (Currently I give a link to the data source and note the data when I've downloaded it. Additionally, I note the date/version of the source if possible.)

Are there projects that try to solve this issue? (e.g. by providing space for important / interesting projects like dblp, a version history and good download speeds)

  • 1
    What about figshare and zenodo? I am advisor of the first one, so I can tell you that they will likely accept larger uploads for free. I depends how much is "several GB" :-) It would be an advantage for you to use such systems, as the data becomes easy to be discovered and can be cited. – dgraziotin Dec 29 '13 at 15:00
  • If you have a few bucks to spend, you could buy some Google drive or DropBox hosting and have a publicly-available link to a few GB of data. The DropBox public folder and the Google Drive both support public links, to my recollection. Drive is a little cheaper to buy data on, but I find Dropbox a little more user-friendly. Others might have services they prefer instead. – Namey Aug 1 '14 at 22:59
2

Both the problems you state are significant, and (I think) very problematic to the further progression of computer science as an actual science. I work in services / software engineering, and quite honestly, reproducibility of results is very, very bad in my work as well as in the work of others.

I will comment on both of your issues, starting with the second one:

The data is either not at all available or it is not clear which version of the data was used

In many cases, this is actually the easier-to-solve issue. I would assume your department has IT resources (e.g., a department web space) that you can use for such purposes. Additionally, there are other, more field-specific, repositories that you can upload data to (for instance, there is the UCI machine learning data set repository or the Grid workload traces archive). Making data available to the public is really only an issue if you are not allowed to do so because of commercial interests of your data provider. For instance, I have access to real traces of executions of business processes of a big german logistics provider, which I am not allowed to give out. This clearly limits the usefulness of these traces to me.

The second problem

The source code that was used to generate the results is not (publicly) available

is sort of tricky in practice. Clearly, throwing your code to Github is easy, but this hardly solves the issue of reproducibility. You will still need to include pretty detailed usage instructions and documentation to make this code of any use to another researcher. This may seem trivial for a naive 1000 Lines-of-Code implementation of an algorithm, but, for instance, my current research prototype approaches 25.000 lines of Java code plus a bit of XML and Groovy. Just putting this up on a repository is not enough, if we consider that a paper can only be considered reproducible if it is possible to do so in reasonable time. An additional problem in my field is that often prototypes are built for a specific execution environment. For instance, my department has a small OpenStack based private cloud, and many demonstrators that our students build realistically only run in this environment without drastic modifications.

I am currently in the transition between a postdoc and a more independent research position, and one of my personal goals is to make all work that I and my students publish easier to reproduce. However, so far we are only making baby steps, but at least we are trying :D

  • One way around the build issue might be to specify a required barebones platform (e.g., a cloud instance) and then provide a build script and an execution script that will pull code and required files from your repositories and 3rd-party sources. Secondly, if your project can work on a Vagrant Box, I have seen this highly recommended by some people I know who worry more about this than I do. – Namey Aug 1 '14 at 23:03
2

Recently (at least) two platforms have appeared which offer web space for sharing scientific data. They provide Digital Object Identifiers (DOI) for uploaded datasets which can be used to refer to the data easily.

  • Figshare seems to be the more prominent option. All data is shared under CC0 license. There is a 1 GB limit for private data and no limit for public data.
  • Zenodo is part of a European research project. You can specify any license you prefer for your dataset. Currently they have a 2 GB limit per file but you can upload as many files as necessary. For larger files, you need to contact the administrators.
  • Although Figshare does not limit the storage of public data, it has a limit of 250MB per file. However, I have seen them lifting the waive for 2-4GB datasets. – dgraziotin Dec 29 '13 at 16:36
  • @dgraziotin Good to know! Feel free to suggest edits to my answer. – underdark Dec 29 '13 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.