26

An article is published and the results depend heavily on a piece of code that the author wrote themselves.

The author has not shared the code with the community, even though they have been using it possibly for years now.

If I request the source code from the author and they refuse to share it, can I "demand" to see it (for example, making a formal request to one of the journals that published one of the articles)?

  • Comments are not for extended discussion; this conversation has been moved to chat. – aeismail Sep 23 '17 at 16:57
  • 3
    @Gabriel: are you a native English speaker? Notice that to demand has not the meaning demander has in French (or similar verbs of most Latin-related languages)! So look in your dictionnary! – Basile Starynkevitch Sep 24 '17 at 7:55
50

I believe you are asking the wrong question.

You are asking: Can one demand to see code used to generate an article?

To which I would answer: Assuming you're in a country with free speech, you can "demand" the code. Just make sure you do not threaten the author or otherwise break the law with your "demand". However, you are unlikely to be successful in forcing the author to share the code with you. A "demand" for code to me would come off as arrogant and I would ignore or decline a "demand".

I think a better question is: How can I get the code from an article I am interested in looking at or using?

To which I would answer: It depends upon your subfield and why the author will not share her or his code. Some subfields now require/expect authors to share code and data. For example, the American Statistical Association recommends sharing code to create reproducible results. Here are the steps I would follow to get the code from an article I wanted:

  1. I would first try reaching out to the author and asking nicely if she or he would be willing to share their code with you.
  2. If the author declined, I would ask the journal editor if they could get the code to support the conclusions article because you wish to reproduce the article's results. Also, some journals now have open code/open data requirements. Check the policy of the journals the author published in.
  3. If you end up in hostile situation with the author, and you think they are being misleading based upon their findings, I would contact the research integrity and ethics office at the author's university (e.g., Illinois or Rochester).
  4. If you truly suspect scientific misconduct as the reason behind not sharing code, you could take this farther by writing a response article questioning their findings and state that they refused to share code with you. For example, watch how this author investigated a cancer scandal. The end result was prison time for the fraudulent scientist.
  5. If the author's funding agency requires open access to code, you could contact them and request they force the author to share their code. For example, all data and code development funded by the US Government is required to be open.

I would suggest working with the intellectual property office or similar office at your institute if step 1 fails. They can guide you on what options, if any, you have.

Last, you might need to accept that you cannot force someone to share his or her code with you.

  • 15
    @Gabriel - Or, of course, you could write your own code to try and reproduce a subset of the results. That might be a useful thing to do in and of itself to understand the issues. – Jon Custer Sep 22 '17 at 15:07
  • Comments are not for extended discussion; this conversation has been moved to chat. – aeismail Sep 23 '17 at 16:55
  • 2
    As I pointed out in an earlier comment that was moved to chat, I believe the claim "For example, all data and code development funded by the US Government is required to be open" in item 5 is false. Specifically, the National Science Foundation does not make such a requirement of researchers it gives funding to. – Dan Romik Sep 23 '17 at 19:37
  • Link to case in #4: laboratoryequipment.com/news/2015/11/… – Stilez Sep 24 '17 at 9:57
  • @DanRomik OMB Memorandum M-13-13 applies to NSF as well as most US funding agencies (expectations include sensitive research such as DOD). The policies and speed of applying the order vary across agencies. The NSF has started moving towards requiring Open Data. Here is the NSF's page outlining their policies. Also, code may or may not be fall under the order depending upon the agencies interpretation of the policy. – Richard Erickson Sep 24 '17 at 21:26
28

The answer to your question is very simple, but it is not the one you are clearly hoping to get: No, you cannot demand (and hope for your demand to be met or to be seen as even remotely appropriate, that is) to see the code the author used, either from the author or the journal where they published.

The reason is also simple to explain. Academics don't make "demands" of each other. The way academia works is that each researcher does their research and releases their work in the way that they think is appropriate, and the community evaluates them and gives them credit for it as it sees fit. Do I think researchers should release their code? Absolutely! I also think journals should make that a condition for any work whose validity depends on the code to be published. But I don't get to decide how journals and other researchers conduct their business.

On the other hand, I do get to decide whether I trust a result described in an article; whether I recommend papers I am asked to referee for publication; who to invite to give a talk at my department or at a conference I'm organizing; who to write a recommendation letter for for their tenure case or job application, etc. And I am also free to express my opinion to others that a result can't be trusted if it can't be verified. That is precisely how things should be, and the result is that people who don't release their code, and their results, will (generally speaking) have a lower level of credibility and lower reputation than those who do.

So, unfortunately you need to accept that you cannot make "demands" or force an author to give you their code when they don't want to. But you can, and should (if you feel that strongly about it), make your voice heard in your research community that you believe releasing the code underlying scientific work is a good practice, should be the norm, and should be a precondition to having the work refereed and/or published. If enough people express such opinions, we will start to see some progress in this increasingly important area.

Edit: here's another way of looking at this issue. Availability of code is just one of many factors that make some research products more useful than others. So I think it may help to frame the question more generally: If an author published a paper that describes their research results, but the description has a flaw that makes it less useful than it might have been otherwise, do I as a reader have a right to demand that the author fix the flaw? For example:

  • A mathematical proof is presented with "some trivial details left to the reader as an exercise" (the author forgets to mention that filling in those details takes 2-3 weeks of frustrating calculations, at the end of which the diligent reader is still uncertain if the result is true);

  • A paper is written by an author who doesn't speak English well, to the extent that reading the paper becomes difficult and in certain parts the author's intent seems impossible to decipher.

  • A physics paper cites certain results from the literature in a vague and imprecise way that makes a mathematically inclined reader wonder which of the claims in the paper are theorems, which are conjectures, which are heuristic derivations not meant to be taken literally, etc. A look at the cited literature does nothing to clear up the confusion.

  • Etc etc.

The answer in all of those cases is again a resounding no: it is completely inappropriate to demand that authors improve their papers (unless you are a referee, and even then you cannot demand it, you can at most make it a condition for recommending publication). Researchers already have every incentive to want to improve their papers as much as they can; if they don't do so, either they are incapable of it, or they can do it but decided that doing something else is a better use of their time. You as a reader need to have the maturity to accept that some papers you read will be useful and trustworthy, and others will be less useful and less trustworthy, and that the world is not going to bend to your demands and adapt any research study to your vision of what the ideal study looks like. That applies to code as much as to any other aspect of the research.

I'll add that after reading more of your thoughts in the comments you posted, I have to say your entitled tone is off-putting to say the least. The fact that you think authors should make available materials that you decided need to be released, and that if they don't do so that represents some kind of moral outrage, makes me personally think somewhat less of you (and it might make others reach similar conclusions). Since you are posting using your own name, I think this should worry you; hopefully it will also cause you to reflect about your unreasonable approach to this issue.

  • Although my mind is obviously made up about this issue (this is very clear, we agree that code should absolutely be released), I was hoping to get some good input. This is good input, so thank you! – Gabriel Sep 22 '17 at 17:11
  • 15
    @Gabriel you're welcome. One more piece of advice: I think you should separate your emotional reaction to the missing code from the logical question of what can be done about it. You seem outraged by the author not releasing their code, and as I said I don't disagree: hiding methods is very suboptimal and hurts scientific research. However, as for what can be done, you write in a comment "I'm not saying it would definitely work" which suggests an extreme level of naivete. Trust me (and others here), making demands will definitely not work, and will be harmful to your own reputation. – Dan Romik Sep 22 '17 at 17:30
  • Separating the emotional reaction is always good advice in Academia :) – Gabriel Sep 22 '17 at 18:03
6

Reproducibility is an important issue, and there is a growing trend to make more raw data available. Many journals actually require the authors to deposit certain kinds of data in public repositories, for example when publishing a paper with an X-ray or NMR structure of a protein, it is very likely that the journal requires the authors to deposit those into the PDB database. There can be an embargo period, but in the end the data has to be public.

I looked at the author guidelines for Nature and Science, and found passages about code in both.

Nature:

An inherent principle of publication is that others should be able to replicate and build upon the authors' published claims. A condition of publication in a Nature journal is that authors are required to make materials, data, code, and associated protocols promptly available to readers without undue qualifications. Any restrictions on the availability of materials or information must be disclosed to the editors at the time of submission. Any restrictions must also be disclosed in the submitted manuscript.

After publication, readers who encounter refusal by the authors to comply with these policies should contact the chief editor of the journal. In cases where editors are unable to resolve a complaint, the journal may refer the matter to the authors' funding institution and/or publish a formal statement of correction, attached online to the publication, stating that readers have been unable to obtain necessary materials to replicate the findings.

Science:

After publication, all data and materials necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science. All computer codes involved in the creation or analysis of data must also be available to any reader of Science. After publication, all reasonable requests for data or materials must be fulfilled. Any restrictions on the availability of data, codes, or materials, including fees and restrictions on original data obtained from other sources must be disclosed to the editors as must any Material Transfer Agreements (MTAs) pertaining to data or materials used or produced in this research, that place constraints on providing these data or materials.

Your first step should in any case be to ask the author politely for the code. If that fails, you can contact the journal, if they have policies similar to the ones I quoted.

  • Good to see that top journals are requiring code to be shared up front. – Gabriel Sep 22 '17 at 17:57
  • 1
    Many journals have similar requirements - as do many granting agencies. However, "code available upon request" seems to be more honored in the breach than in the observance. It probably helps a lot to convey a convincing reason to the code author why it is in their self-interest to provide you with the code! (e.g., potential collaboration, validation against a common issue?) – AJK Sep 22 '17 at 20:07
3

If an article is published which depends on software which cannot be run the results will be difficult to reproduce. The best case is you implement comparable software and eventually get the same/better results.

The worst case is when the interesting results are the consequence of an undiagnosed error in the code and will never be reproducible. There is no reason to expect the error rate in unreleased software to be better than in peer reviewed or commercially supported software.

Publishing without the code is much like publishing results without showing mathematics, data or methodology. The most reasonable response I know of to this is to disregard the paper.

2

One problem is: even if the authors are happy to share the code, you may not be able to run it. I know a very successful software that generated dozens of papers, and came as third place in a prestigious competition. It works like the following:

  • Run a Docker container which maps one folder to the folder of the host machine.
  • The Docker image is more than 64GB. Nobody in the lab know (have enough effort) to install all the tools from scratch. So they use the Docker image.
  • Make a new folder for the program to be tested.
  • Copy some tools/data from the host machine (via the mapping folder), some tools/data from other folder within the Docker container.
  • The tool itself is a combination of bash scripts and Python. It invokes different tools that were copied and outputs to different folders. Some folders are in the home folder of particular user (the intern who developed this part).
  • The main script to start the tool has 7 parameters.

The most interesting thing is that: there is no documentation (who has time for that). So even if the authors give you the source code, you will not be able to run it. In this case:

  • Should you demand the authors to make their software to run in all machines?
  • Should you demand the authors to write documentation?
  • Should you demand the authors to help you to run their tool? Considering how much effort to spend on each request if this is possible.

What is the protocol to share the source code, considering IP, copyright etc? To open source a tool is not easy. My PI intended to open source a tool built by our group. After several meetings with 4 lawyers of the university over a year, they gave up.

  • 4
    @Gabriel "If you could run it, then someone else should be able to run it too". You don't understand the example I gave. – qsp Sep 22 '17 at 23:30
  • 2
    @Gabriel If you understand the example I gave, you know that it doesn't run outside the host machine. Do you want the host machine itself? or you want to fix a 3000-line bash script that move folders back and forth between host machine and Docker container? By "at least describe in details", are you asking for detailed documentation? If you can run whatever source code people can give you, you are definitely a genius. – qsp Sep 22 '17 at 23:50
  • 3
    @Gabriel: I believe nobody has this intention, because the authors themselves are the first to suffer from it. My point is many (academic) software are just very difficult to run, in particular when they are written by researchers, not industrial engineers. And you can't just demand them to write good programs, good documentation for you to be able to replicate their results etc. – qsp Sep 23 '17 at 0:06
  • 5
    @Gabriel Good faith doesn't guarantee anything. Some researchers are simply bad engineers, and produced bad code, horrible set up. You can't just demand them to write code that others could use. – qsp Sep 23 '17 at 0:26
  • 8
    +1. I work in industry, and we use automated builds on dedicated build servers, try to write maintainable code, etc., and it took me months to compile all our code and lists of tools together to meet a contract's escrow requirement. If you don't design for public release from the very beginning it can be very difficult to bolt it on later. – drewbenn Sep 23 '17 at 0:33
0

I'm actually pretty sure that sharing some particular codes used to obtain certain results is illegal, depending on the method used. Not always the code will be fully developed by the author her/himself. Sometimes people use external libraries which provide numerical methods for evaluating inputs.

I'm keeping it short with some examples from experiences of my own:

At the university I work in, we are provided with some libraries which are paid, and, in a couple occasions, in order to execute some of my codes efficiently, I had to recur to those libraries. There were situations where I used RK4 method ready-to-use subroutines to solve certain problems (eg. RK4 from Numerical Recipes), subroutines which I'd not have the time nor patience to develop my own, given it's not the focus of my research. Could I write a similar code for the purpose? Of course. Would it be pleasant and/or useful? Not at all. Other people have already developed much faster and more precise algorithms, and those were sold to us by them, by a considerable amount of money.

This example was probably somewhat silly, given that RK4 is such a group of simple methods, but there are much more sophisticated subroutines which would require special dedication and knowledge, making it highly unlikely that someone from another field would waste their time learning how or even have the ability to program such thing.

Edit: I think I probably didn't make myself clear at saying some people use libraries bought by the University inside the code, because it might sound like we just call one or another subroutine in one or another line of the code. That's not how it works. Calling subroutines and calculating external functions require a lot of time, which, most of the time, are "copied and pasted" inside the scopes of the author's code, and modified according to the author's own purpose. There is rarely an explicit library import and subroutine call, simply.

As pointed out below in the comments, it isn't really that simple to just let your code available for the public and don't worry about any legal issues.

  • 2
    You can certainly share your code calling the libraries. If then one wishes to reproduce your results can install, buying if necessary, whatever libraries are needed. – Massimo Ortolano Sep 22 '17 at 22:03
  • Sure, but we are also free to copy modify those libraries inside our own code's scopes, since the original code is exposed to us, and I think in this specific occasion, a sue from the seller would fit. – a-sf-d Sep 22 '17 at 22:08
  • 1
    Nobody is asking you to do anything illegal. If you wrote your code around a closed source library, share your code as Massimo said. If you modified the library, share your modifications. – Gabriel Sep 22 '17 at 22:45
  • 3
    I think this is worth more than -4. Certainly it calls attention to the fact that code sharing can require a significant investment of time. (Double-checking legality, stripping out libraries that can't be redistributed, describing the modifications to a library, etc.) – AJK Sep 23 '17 at 2:08
  • 1
    Well, one publishes a detailed paper describing the theoretical model used, expliciting almost every step taken, except that authors don't explicitly show their code, which is a way for anyone who really understood the theoretical model to reproduce the results themselves. I don't think the way articles are published nowadays ask for good faith of readers. If someone fails to reproduce the results, I think they don't understand the theoretical background at all. And, if they do and the results don't match, they should publish a letter showing the correct results by themselves. – a-sf-d Sep 23 '17 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.