The topic of reproducible research is attracting quite some press these days, yet much remains to be done. In this spirit, I am asking what can be a reviewer's role in this — I'll explain fully below.
Like many people, I would like to see academic research moving toward a more “open data” model, especially because the principle of reproducibility of research is central to the scientific method. However, I know that at least in my field (theoretical chemistry), the usual standards are pretty lax. I'll give two examples:
- If you develop your own simulation/modeling code, you do not have to make it public in order to publish your results.
- If you use an existing modeling code (available for free or commercial), you do not need to include your full/raw input files with your publication.
When I peer-reviewed papers for publication in the past, I typically did not ask for this, because (a) maybe my standards are not other people's standards, and (b) the role of the reviewer is more to advise on the quality of the science and analysis of the results.
But, over time, I'm not really satisfied with this approach any more. So: as a reviewer, how much information do you think is reasonable to request from the authors? Should you follow the customs and unwritten standards of your field, or is it okay to push it toward the direction you'd like to see it go? And how much can you push?