24

The topic of reproducible research is attracting quite some press these days, yet much remains to be done. In this spirit, I am asking what can be a reviewer's role in this — I'll explain fully below.

Like many people, I would like to see academic research moving toward a more “open data” model, especially because the principle of reproducibility of research is central to the scientific method. However, I know that at least in my field (theoretical chemistry), the usual standards are pretty lax. I'll give two examples:

  • If you develop your own simulation/modeling code, you do not have to make it public in order to publish your results.
  • If you use an existing modeling code (available for free or commercial), you do not need to include your full/raw input files with your publication.

When I peer-reviewed papers for publication in the past, I typically did not ask for this, because (a) maybe my standards are not other people's standards, and (b) the role of the reviewer is more to advise on the quality of the science and analysis of the results.

But, over time, I'm not really satisfied with this approach any more. So: as a reviewer, how much information do you think is reasonable to request from the authors? Should you follow the customs and unwritten standards of your field, or is it okay to push it toward the direction you'd like to see it go? And how much can you push?

  • 5
    Scientific software does not compute – StasK Oct 23 '13 at 23:19
  • 4
    To follow from StasK's comment, a lot of code written by academics requires significant knowledge of the subject and the code's architecture to use. It is poorly documented and needs careful use to get realistic results. Until we can trust someone else to use it, then we won't release it for fear of bad results damaging our reputation. – Moriarty Oct 24 '13 at 0:29
  • Not an answer, but this article might be of interest: climateknowledge.org/figures/… – Flyto Oct 26 '13 at 7:56
  • There is another hidden issue to watch out for. Many people have some sort of analytics set up for their website, so if you download their code they will see in their log a connection from $YOUR_IP and $YOUR_UNIVERSITY. This may reveal your identity as a reviewer. Maybe the best course of action is asking the editor to download the code and send it to you. – Federico Poloni Oct 26 '13 at 11:27
  • 4
    @Moriarty you should check out the top ten reasons not to share your code (and why you should anyway) on a recent SIAM news. It makes a very convincing rebuttal of your point. – Federico Poloni Oct 26 '13 at 11:29
23

If there are other reasons to reject the paper, then it's certainly unnecessary to request code/data. If the paper looks like something that might be accepted, then you should ask yourself:

Can I certify the correctness and significance of this work, to the necessary degree, with the information that is available?

Of course, the key is the phrase to the necessary degree. To make things more concrete, you might consider:

Would I feel comfortable if the whole world knew that I refereed and recommended acceptance of this paper?

If the answer is no, I don't have enough confidence in the results without seeing the raw data/code, then you should ask for it. You're really doing the authors a favor here -- giving them the chance to convince you by providing additional evidence.

I would be very polite and make the request through the editor. If the code and data are not forthcoming, you should probably say in your recommendation something like

I find the results in this paper compelling if they are correct, but I cannot recommend it for publication without verifying the data/code that underlies those results.

Of course, the degree to which a referee is expected to verify the correctness of results varies greatly between fields. But you can always choose a personal standard higher than what's usual in your field. Just realize that good refereeing takes a significant time investment.

3

Here are my two cents.

Yes, if there is code, you should ask for it (if possible). If you get it, then test it. Bear in mind if the authors don't have usable code, they probably aren't going to give it to you, but it doesn't hurt to try.

If the authors have provided code, the authors will probably be delighted if you test it. Most of the time people don't bother. They'd probably be seriously thrilled if you provide useful feedback and suggestions for improvements. This practically never happens. Personally, if an academic reviewer was to provide useful feedback on some code I had written, I'd think I was dreaming.

In particular, if you can't reproduce the results using the code, then I suggest documenting your reproduction difficulties in the review. I believe that such a failure is not considered a dealbreaker in academic research, because the ideas are the most important thing. However, it is still nice for everyone when this is possible.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.