15

It is still relatively uncommon for social scientists to share data or code as a part of the peer review process. I feel that this practice runs contrary to notions of replicability and reproducibility and have a desire to voice opposition to instances in which manuscripts are submitted without data and code.

Where, however, is such opposition appropriately expressed?

I am specifically curious about whether or not it is appropriate to refuse to review an article in the absence of code or data.

UPDATE

I'm going to watch this for a couple of days to see what surfaces. FYI I cross-posted this to Andrew Gelman (http://andrewgelman.com/2015/09/14/its-not-so-easy-to-share-data-and-code-and-there-are-lots-of-bureaucrats-who-spend-their-time-making-it-even-more-difficult/) for those who are interested in seeing his response and the response of his followers...

Improve this question
6
  • 1
    What is the publication's editorial policy in this regard? What does the editor say? If you aren't on the editorial board, you have no vote except to stop buying/reading that publication. You could try a letter to the editor, but....
    – keshlam
    Sep 14, 2015 at 23:44
  • 5
    @keshlam: If you are a reviewer, you certainly have the power to decline to review a paper if you don't feel you have enough information to properly evaluate it. Doesn't matter what the journal's policy is.
    – Nate Eldredge
    Sep 15, 2015 at 0:25
  • You can always decline to review, or reject... but that may just mean they find another reviewer, if they don't agree with you.
    – keshlam
    Sep 15, 2015 at 1:36
  • 2
    FYI: Are there any journals or conferences that take into account the availability of the source code when selecting the papers to publish?
    – Franck Dernoncourt
    Sep 15, 2015 at 3:57
  • 1
    @NateEldredge Editorial policy generally states what it considers to good way presenting data, which details should be included etc. If the reviewer totally ignores this policy, he is rather unethical and unprofessional. The reviewers job is to judge the merits of the manuscript BASED ON EDITORIAL GUIDELINES. If he doesn't like those guideline, should feel free to reject the invitation for reviewing for a journal he obviously considers sub-par.
    – Greg
    Sep 15, 2015 at 4:27

3 Answers 3

Reset to default
7

I would suggest first spending a little bit of time reflecting to yourself on why you want to see sharing of code and data, and which code and data you would want to be shared. Is it more ideological in nature ("information wants to be free"), or at its root is it a belief more about scientific ethics and efficacy?

I am almost certain that you do not actually want all data and code to be shared. For example, you probably do not want identifying information about subjects to be shared. Likewise, you probably do not care too much about little bits of utility code that don't have anything much to do with the science. Where do you draw your boundaries and why?

The reason I would suggest doing that, is that you will then be in a good position, as a reviewer, to persuasively argue for the inclusion of code and/or data for objective reasons that the editor and author are more likely to agree with, even if their personal preferences are different. I don't think you'll have much effect by refusing to review (unless you can get a lot of colleagues to exert pressure along with you), but I think you may make much more difference by refusing to accept poorly justified conclusions in the absence of result and method information to support them.

Improve this answer
3
  • 3
    I would also suggest that requiring any support for said code would immediately unblind the peer review process at worst, and at best make the journal a tech support routing system and slow things down considerable.
    – Fomite
    Sep 15, 2015 at 4:51
  • 1
    @Fomite Perhaps my phrasing was unclear: I meant "support" not as in "tech support" but as in "evidence that supports the conclusion"
    – jakebeal
    Sep 15, 2015 at 11:34
  • It appears I was being unclear - I was adding a reason. For example, where do you draw the boundaries also includes "Does this have to work without any back and forth with the code authors?" etc.
    – Fomite
    Sep 16, 2015 at 4:17
1

Strategically, you might consider that you'll probably have much more impact if you become the editor of a journal. That way you'll be involved in setting editorial policy, and more importantly, enforcing that policy. Refusing to review papers will not help you become editor.

This presumes that you care enough about this issue that you want to spent the time and effort that comes with being a journal editor. But there are more reasons for becoming an editor then just this one issue, so I would at least consider it.

Improve this answer
1

This is a very UK specific answer.

The UK Research Councils have principles on open data and policies on open access. In the most recent clarification this also covers software. In essence, RCUK funded research should have the data and software essential for reproducibility released before publication.

My approach to this is that

  • when refereeing UK grant proposals I always check that the grant explicitly mentions they're following these policies, and if they don't explicitly say how, I raise that as a question;
  • when refereeing papers that cite support from RCUK grants I always query if I can't find the software/data that allows them to follow these policies.

In both cases I expect the impact to depend heavily on the preferences of the panel/editor (I don't think I've seen much as yet).

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .