As a reviewer, how much raw data or code should you request?

Question

The topic of reproducible research is attracting quite some press these days, yet much remains to be done. In this spirit, I am asking what can be a reviewer's role in this — I'll explain fully below.

Like many people, I would like to see academic research moving toward a more “open data” model, especially because the principle of reproducibility of research is central to the scientific method. However, I know that at least in my field (theoretical chemistry), the usual standards are pretty lax. I'll give two examples:

• If you develop your own simulation/modeling code, you do not have to make it public in order to publish your results.
• If you use an existing modeling code (available for free or commercial), you do not need to include your full/raw input files with your publication.

When I peer-reviewed papers for publication in the past, I typically did not ask for this, because (a) maybe my standards are not other people's standards, and (b) the role of the reviewer is more to advise on the quality of the science and analysis of the results.

But, over time, I'm not really satisfied with this approach any more. So: as a reviewer, how much information do you think is reasonable to request from the authors? Should you follow the customs and unwritten standards of your field, or is it okay to push it toward the direction you'd like to see it go? And how much can you push?

Answer 1

If there are other reasons to reject the paper, then it's certainly unnecessary to request code/data. If the paper looks like something that might be accepted, then you should ask yourself:

Can I certify the correctness and significance of this work, to the necessary degree, with the information that is available?

Of course, the key is the phrase to the necessary degree. To make things more concrete, you might consider:

Would I feel comfortable if the whole world knew that I refereed and recommended acceptance of this paper?

If the answer is no, I don't have enough confidence in the results without seeing the raw data/code, then you should ask for it. You're really doing the authors a favor here -- giving them the chance to convince you by providing additional evidence.

I would be very polite and make the request through the editor. If the code and data are not forthcoming, you should probably say in your recommendation something like

I find the results in this paper compelling if they are correct, but I cannot recommend it for publication without verifying the data/code that underlies those results.

Of course, the degree to which a referee is expected to verify the correctness of results varies greatly between fields. But you can always choose a personal standard higher than what's usual in your field. Just realize that good refereeing takes a significant time investment.

Answer 2

Here are my two cents.

Yes, if there is code, you should ask for it (if possible). If you get it, then test it. Bear in mind if the authors don't have usable code, they probably aren't going to give it to you, but it doesn't hurt to try.

If the authors have provided code, the authors will probably be delighted if you test it. Most of the time people don't bother. They'd probably be seriously thrilled if you provide useful feedback and suggestions for improvements. This practically never happens. Personally, if an academic reviewer was to provide useful feedback on some code I had written, I'd think I was dreaming.

In particular, if you can't reproduce the results using the code, then I suggest documenting your reproduction difficulties in the review. I believe that such a failure is not considered a dealbreaker in academic research, because the ideas are the most important thing. However, it is still nice for everyone when this is possible.