5

Since reproducibility is a major principle or basic norm of scientific experiments, simulations, and computations, what would you do or what to do if the author of a published CS article does not respond to code requests?

4
  • 3
    It could be that they are busy, especially with putting teaching material on-line because of COVID. It could also be that your emails are ending up in their spam filter. I'd also suggest that reproducibility means you can reproduce their results from the information provided, which may not mean code. Just running their code is not a good test of whether the method is right. – Dikran Marsupial Jan 12 at 8:55
  • 8
    Why don't you yourself replicate and implement the method? – Roboticist Jan 12 at 9:10
  • 2
    It is also possible if the paper is not recent that they don't actually have the source code, especially if it was the work of a PhD student, for example. I don't have source code for all of the papers I have written (and research code seems more vulnerable to "bit-rot" than most, so it may not even run without substantial work). – Dikran Marsupial Jan 12 at 9:20
  • 9
    I do not think anybody has an obligation to you to respond to code requests. – lalala Jan 12 at 18:51
14

Simply nothing, but trying to implement it yourself!

According to my experience, there would be four potential reasons because of which authors may refuse to share their code:

1- Their code needs to process some particular data that they are not allowed to share. Thus, the code by itself, without that specific data, does not reproduce the results of the paper. So, sharing the code does not fulfil any reproducibility purpose.

2- They feel that they may still release more "major" contributions of that code. Then, they would postpone the public release of the code until they feel that the code is milked enough, and they have run out of further remarkable ideas.

3- They may believe that if one is really interested in their work and the cultivation of ideas to expand its results, she herself codifies the paper since personal codification often leads to way deeper levels of understanding of what's behind a piece of code, rather than just running some already-cooked snippets.

4- They don't have further access to the code. Not everyone may keep a clone of her contributions on her personal hard drive depending on the policies of the academic or industrial body under which that code was written.

2
  • 8
    +1, another issue is that the cost of providing code is not zero. Users will want support and they are not always very polite about it (or willing to read instructions first). There are no institutional incentives for researchers to provide support or to write code to a better quality than the minimum required for the paper, rather a disincentive as this increases the time to the next grant submission. – Dikran Marsupial Jan 12 at 9:37
  • 1
    And if you reproduce an interesting result, you may publish your work in ReScience C! rescience.github.io – Matthieu Latapy Jan 12 at 19:12
2

Executing the same code on the same or similar data can be expected to lead to the same result. It isn't reproducible science.

The proper way to think of it is to think about what scientific question is being asked and find a way to answer it, either positively or negatively. In other words, from the nature of the question, create a methodology sufficient to give appropriate evidence for the truth or falseness of the hypothesis. Then execute that methodology in an appropriate way and extract the results, whether they confirm the original study or not.

But simple repetition isn't reproducibility.

It would be a different question if you were exploring whether the code itself is flawed and leads to biased results. But the same is true for questions about methodology in general. Likewise if you suspected fraud in the presentation of results.

But for true reproducibility conduct an independent study of the question at hand.

0

Some journals, especially in the physical sciences, require code or data to be made available as a condition of publication, either upon publication or upon request. For instance, Proceedings of the National Academy of Sciences has this statement:

Authors must make Unique Materials (e.g., cloned DNAs; antibodies; bacterial, animal, or plant cells; viruses; and algorithms and computer codes) promptly available on request by qualified researchers for their own use. Failure to comply will preclude future publication in the journal.

You could try contacting the editor of the journal if the article was published in such a journal, who would pressure the authors to make their code available. However, I would first try contacting other co-authors (first rather than senior, or vice-versa) who might be more willing or able to respond to the request.

4
  • 2
    Hmmm, why would an editor "pressure" an author of an already reviewed and published article if it isn't required by the journal or norms of the field? I think the policies you name are intended to prevent fraud and misconduct. Especially in health related fields. – Buffy Jan 12 at 20:13
  • @Buffy Because the editor might care if the paper published is fake or not? – Yakk Jan 12 at 21:28
  • 2
    @Yakk, It's been through the editor's own review process. If you allege fraud, don't ask for the code, allege fraud. – Buffy Jan 12 at 21:34
  • @Buffy in the case of "Proceedings", the editor would pressure the authors because sharing code is a policy that all authors must agree to as a condition of publication. If the journal in question doesn't have this policy, then there is no cause to contact the editor. – Andrew M Jan 14 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.