Is it appropriate to mention that authors refused to provide code samples when asked?

Question

I work in a sub field of computer science. There was a recent paper (2020) that was published in a reputable conference that claimed to achieve state of the art performance on a specific data set. This data set is a hot dataset, and my research group has many papers on that specific data set (improving state of the art). I asked the authors very politely if they can share their code samples for their fairly complicated algorithm, to which they rejected and told me that it is not available.

One of my colleagues took 2 weeks to code up the algorithm, and verified its performance, as well as the claimed performance. Meanwhile, we reached a new breakthrough on the dataset and achieved a new state of the art. We are currently writing a draft.

Now, I can't help but be left with a bad taste from this whole experience. I don't see why they wouldn't share their code especially if they claim a state of the art on a very important data set. Most papers have to be compared with the state of the art to some extent, and so essentially, they're slowing down all researchers in this specific field to recreate their own work prior to publishing their own research.

Now my question is, would be advisable to add a note that the implementation of THEIR algorithm is our implementation, and that we reached out to them for their code and they declined?

Answer 1

would be advisable to add a note that the implementation of THEIR algorithm is our implementation

Absolutely yes. This provides important context for your experimentation and as such, is valuable information for the reader.

Even better, you could make your implementation of their algorithm publicly available, so that future groups of authors will not encounter the same situation as you did (and will cite your paper).

and that we reached out to them for their code and they declined?

That is the more delicate question to ask. It would surely be legitimate to do so, as it would represent the situation accurately. But such a statement has a social implication, which is that the authors appear in a slightly negative light (perhaps rightfully so). Proceed with caution.

Answer 2

As a software engineer, I'll give the dissenting opinion.

Source code is not an algorithm. It's a "dusty mirror" version of something which hopefully is the algorithm they intended and which hopefully performs correctly. Software being software, and coders being human, there are many ways in which those "hopefully" parts may not be as expected. It is even possible (albeit unlikely) that the perceived improvement may come from an unintended feature of the implementation, and/or that perceived improvement may be as a result of making an implementation trade-off which has deleterious effects elsewhere.

In general science terms, the algorithm is the hypothesis you're testing; and the source code is your laboratory glassware, circuit boards, interferometer, or whatever other equipment you'd use to test the hypothesis. A general principle of science is that reproduction of results must be independent. That means you don't just get a second person to blindly run the same tests on the same glassware with the same bottles of reagents - instead an entirely separate team reproduces those results from first principles to confirm them. Independent testing of a hypothesis is what moves this from merely a hypothesis to a solid theory.

By implementing the algorithm independently, you have performed a vital scientific role in reproducing the original results. If the original authors had simply given you the source code, there would have been no reproduction and test of the algorithm. If they'd given you the source code, you might as well count photocopying the paper as "reproducing their results". :)

You would like to push ahead with the state of the art, and that's great - it shows that you're ambitious in your field. But independently verifying a hypothesis to be able to call this a "theory" is not a waste of time, and slowing down to get this groundwork in place is not something which should leave you with a bad taste if the "science" part of "computer science" is for real. Otherwise you're not doing science, you're doing hacking; which is valuable in its own way, but it isn't science.

Answer 3

You can, and maybe you should in order to make your position clear that you think that academic codes should be shared if that is feasible. Many scientists will agree that sharing codes leads to better science because it makes it easier to replicate research, and also to find mistakes.

At the same time, not everyone feels like that, and you happen to have found someone of that opinion. To publicly out them as "backward" and "unwilling to make it easier for others to replicate their findings" may be ethically the right thing to do if you believe in open software, but it may also rub them the wrong way and create "a bad taste" in their mouths to use your phrase.

So the question is: How much do you want to insist on principle, and how much do you potentially want to alienate one of your colleagues? That question likely comes down to where you and where they are in your respective careers.

Answer 4

My advise is to get the most out of the situation as it is: you did a proper replication from the description in the paper (as opposed to "just" running their code) and verified their claims. Publish this as such: this is proper scientific work.

In your own interest, I'd not mention that you did not do this replication entirely of your own free will.

---

TLDR:

I fully agree that it is not nice(TM) to not share code (see also below).

According to what you describe, however, this inavailability of the code has lead to further scientific advance than had the code been available to you:
You verfied their claims by implementing from scratch the algorithm described in their paper without any access to their code. It is scientifically relevant to describe this in your paper since it means that you provide the next higher level of reproduction compared to "only" running their code on your computers: you also verified that the description in the paper is sufficient to reproduce the claims.

IMHO this is particularly important since we do have a reproducibility crisis in many fields.

Sharing code helps with what would be called repeatability in my field (analytical chemistry), but reprodcution or replication of a study/paper is more than that.

Yes, code sharing is nice since it saves a whole lot of work for those who don't want to replicate, it is often the easiest way to unambiguously describe what computations were done, and is also very helpful in tracking down misconceptions by the reader. Still, iff code sharing leads to everyone (or too many) skipping the effort put into proper reproduction/replication it may be counterproductive wrt. the reproducibility crisis.

Thus, I'm a bit torn:

• Their not sharing code is not nice.
• Saying that you did the implementation from scratch only because the code was not available also means admitting on your side that you'd skip a more thorough verification of that paper if you have a choice. Also not nice.

The ideal situation would have been had OP's group done a full verification of the paper and have gotten access to the paper authors' code.

Personally, I'd not allude to any non-scientific reason for tackling the reproduction of that paper.

I use "not nice" here to indicate behaviour that is less than ideal (for scientific advancement) but is also clearly within the allowed range of behaviour. To be clear, I don't want to force anyone into reproducing other groups' studies. At the same time, if all groups deny to do this kind of "homework", the risk for replication crises increases.

---

There are levels of not sharing that I consider definitively problematic from a scientific point of view:

• Your proper efforts cannot reproduce the claims and the authors refuse to guide you (whether by sharing code or by looking at your code and telling you what needs to be done differently)
• They claim in the paper that they will share the code, or the journal guidelines stipulate that they'll share but they refuse.

So far, the question doesn't indicate any of these points apply.

---

To chime in with Anonymous Physicist's answer: I have been working at a research institute where the very official line was that neither code nor data are published with the paper ("are available on reasonable request" was the official wording) - that I would have liked to not only show my research code but put it under an open license did not matter at all.
Thus, had I been author of said paper, you'd have gotten a polite reply that I cannot give you the code without official administrative permission, and would you please contact the director about it. However, I can easily imagine institutes specifying that whoever is contacted about sharing the code has to reply in the negative without creating further work to upper management...

From this experience: if you critizise the authors for not sending the code please make sure that you attach the blame to the correct persons.

Answer 5

Without knowing why the authors declined, it's a bit hard to tell if the other authors were "nice" or "naughty".

There are some legal and institutional pitfalls when it comes to sharing code. As an example, the university may have the right to check if the concrete research output contains patentable material. The authors may be forbidden from sharing their code if this check is ongoing.

So I suggest avoiding putting a negative light on this interaction if possible. Yet, it is important for the reader to know that you wrote your own implementation. And it's also important for the reviewer as well, as the reviewer may think that the other approach looks unnecessarily bad because you didn't implement it well.

Many papers have an appendix that will not be published with the rest of the paper. You could add a note for the reviewers that sharing the code with you has been refused for unknown reason, so you had to implement your own version. If you want to mention in the main part of the paper why you didn't use the original implementation of the other approach, you could mention that it was unavailable, hence you implemented your own version and made sure that you took similar care in both implementations not to miss important optimizations.

Answer 6

I don't see why they wouldn't share their code especially if they claim a state of the art on a very important data set.

Hypothetically, one can make software that can't legally be distributed under copyright law.

If I own a book I can write notes on it and cross out sections all I want - but I can't distribute copies of my modified book. Likewise, if I want to extend a copyrighted Matlab library in a way the authors didn't anticipate, I can copy-and-paste from it and modify it as much as I like for my own use - but I can't distribute the resulting derivative work.

If you want to assume good faith, you could assume some similar benign reason on their part.

Of course, a cynic would say authors are rewarded for wringing as many papers as possible out of a piece of work, and if there are obvious follow-on papers (like obvious tweaks, or tests on other datasets) not releasing the code until you've written those papers is the behaviour their paymasters reward.

Answer 7

With the information given, it is not possible to assess the motivation of the original authors. As a frequent publisher of open source scientific code, I can think of several.

1. They did not want anyone to replicate their results using their code, as they wanted to make more publications from it (certainly not nice, but seen before).
2. Their code was not in a state to be used be anyone else (PhD-ware), and they don't have the man-power to clean it up.
3. They make use of proprietary libraries, or have internal licensing issues, not allowing them to share the code.
4. They simply don't want to be bothered by the inevitable tech support issues, that comes with supplying ones code to other people.

And the list could be continued.

You can only act given the information at hand, and that is, that at the moment the code does not exist in an open form. Fortunately, you were able to reproduce the results, and you are now free to publish your own implementation along with your article, prompting users to cite your paper, as well as the original one. After all, you have made a significant piece of scientific work, which should now be credited.

You don't need to be snarky in your paper. Write about what you did, and not what other people did not. You can note that:

The implementation outlined in this paper reproduces the performance by author A, quoted in ref. [A]. The present implementation can be obtained from XX, allowing for further independent validation.

Answer 8

Unless the topic of your research is open source software, your journal article is not the right place to criticise your colleagues for not releasing their source code. Journal articles and conference papers should be about the knowledge you have created, not your opinion of your colleagues.

Before criticising your colleagues in any venue, you should consider if releasing the source is a reasonable option for them. There may be legal restrictions you do not know about, or a very inconvenient administrative process.

Peer review is a great time to encourage transparency.

Answer 9

Is it appropriate to mention that authors refused to provide code samples when asked?

I work in a sub field of computer science. There was a recent paper (2020) that was published in a reputable conference that claimed to achieve state of the art performance on a specific data set.

There are several issues.

FWIW, I am a semi-academic research engineer in France, working at CEA LIST in a cybersecurity lab, where I develop Bismon (a GPLv3+ software funded by two H2020 projects: CHARIOT and DECODER and described in this draft report)

• how was the work described in the paper funded? if the funding agency is military (e.g. DARPA) the funding contracts may require academic publications but disallow any publication of the code or of the data set. Even H2020 funding may forbid publishing of data, or simply getting permission to publish the data or the code may require a lot of red tape work that usually is boring for any academics.

• who owns that dataset and how was it funded? If the dataset was difficult to collect (think of biological data, e.g. Covid19 related, or of nuclear industry related data) you might not be able to access it

• are there any privacy issues or personal data issues? Regulations like GDPR are then highly relevant

• what legal system and copyright applies to both the data and the code samples? Read also Jean Tirole's work, in particular his paper with Josh Lerner on simple economics of open source. Notice that Shoshana Zuboff books and papers (e.g. Big Other: Surveillance Capitalism and the Prospects of an Information Civilization for example) are also relevant.

• the dataset or the code may be an asset for future research grants.

Notice that academia is a small world (read also about Ershov number and more importantly Erdös number). Mentioning in a formal written paper that some authors refused to provide code samples may backfire, and could perhaps impact negatively your academic career in the next few years. The common practice is dual: mention in acknowledgment sections the people and their institution who provided you with code and data.

If publishing the code or the dataset was not previously and explicitly mentioned in the research grants funding the work, it is likely that you might not get it easily. I got the habit of mentioning (at least in footnotes) such things in most research grant proposals I contributed writing.

By personal experience, obtaining permission -within professional context- to publish some code as open source or some dataset as open data is very boring and stressful (and nearly impossible after the work has started). You should mention these things in writing in research proposals and grants.

there is no such thing as a free lunch

See also these slides, and the RefPerSys project. So far, funding is the main problem in it.

In Europe, ITEA3 and BDVA and HorizonEurope could give interesting contacts. You should find the equivalent in your place.

Answer 10

Some ramblings -just to provide another perspective- before the answer.

I can share my experience as a "user" for similar occasions. CS is not my field, and I can not rewrite each and every algorithm that claims to be a "drastic improvement" (Spoiler alert: Most are not). I simply do not have time, and brute force algorithms usually work for me. Usually. But if I had access to said algorithms (read: actual code), I would have used them and that would be a huge enabler for my future research.

Some answers/comments claim, roughly stating, using the same code is not reproduction. For CS, this might be true to some extent -I won't even argue that. But reproduction is the real concern here? I value usability more than pure algorithm reproduction.

There are other fields (such as mine, sports sciences) which would use some "help". Don't take it wrong way, I can most probably code said state of the art code myself, given the algorithm. But it simply is not worth my time. Because a potential (and a very little one in my past experiences) improvement in execution speed/memory footprint does not justify working more on my already-working proof-of-concept code than tackling the next problem on my list.

There are lots of articles proposing various algorithms, but not bother to even explain them properly.

Back to your question:

Now my question is, would be advisable to add a note that the implementation of THEIR algorithm is our implementation, and that we reached out to them for their code and they declined?

Definitely advisable for the first part, since that is pure blind reproduction of their algorithm, and it adds value to both parties' research.**

For their refusal: How can you be sure that their proposed algorithm was not a product of a lunch-time conversation, and never before realized in actual code until you did it? How can you be sure their claimed performance figures were not fabricated? You simply trust the researchers? What if your implementation did not verify their algorithm's superiority? Would you still publish it, or give in to appeal to authority? Would you check -rigoriously- if their code match up with their explanation of the algorithm: i.e. there is nothing up their sleeves?

I believe these questions are the reason their refusal leaves a bad taste; not your coworker's 2 weeks.

Whether in a publication/presentation or in your circle, I think you owe science to make this info public. The latter seems less risky if you are not "well established yet" though.

Answer 11

The whole providing source code of the paper thing has been discussed to death on this site and elsewhere, so I won't get into it much. I agree with you that they should have given source, and it should be required by journals anyhow. That said, currently it is not. It would be nice if they helped you reproduce their work, but they don't have to.

If you had discovered that when you take their instructions at face value the results are not reproducible, you would have cause to write to the editor of the journal and the onus would be on them to help demonstrate their own findings as legitimate. However, it sounds like you[r friend] were able to reproduce it without their help, you just wish it was easier. Fair enough, but you can hardly demand they go out of their way to make your life easier.

If you intend to distribute the code you wrote, there is nothing wrong with that and it's a very good idea. I wouldn't attach a little manifesto saying the authors were unhelpful jerks and you wrote this prove your point. While it is true and you are within your rights, it will make you look bad. It's worth explaining that your implementation is independent and not based on the original code, so as to not confuse any readers, but there is no need to go into why they didn't share or who is at fault -- these are not matters of fact pertaining to the research itself. You can say something like "this code is an independent implementation of the algorithm in Smith et al. 2020, written without access to the authors' own implementation". That last bit you can fine tune depending on how fine a point you feel like putting on it. But, once again the less you say about how much you dislike that the authors wouldn't share the code with you, the better (in the long run).

I don't see why they wouldn't share their code especially if they claim a state of the art on a very important data set. Most papers have to be compared with the state of the art to some extent, and so essentially, they're slowing down all researchers in this specific field to recreate their own work prior to publishing their own research.

There can be many reasons, not all of them nefarious.

• If the journal doesn't require code, providing it is extra work and inefficient.
• You would have to document how other people can build and run it on their own computer - something which can take extra work and maybe is not a concern on your own system.
• Generally the code is not that helpful if not portable - while the algorithm is already portable.
• Just because they provide the code doesn't mean it will be useful to you. It could be in a language you don't know.
• The code may be messy and perhaps they don't feel like inviting pointless questions. This may seem dubious but imagine you were asked to provide every single draft and napkin drawing, and then people made you explain random thoughts that were jotted in the margin and were abandoned later on anyhow.
• Whenever you provide code, all sorts of people start asking you for help with matters that are sometimes barely connected to the work itself. While you should be available to answer questions about your work, answering questions takes time. If the journal didn't require it, why create additional work for yourself?

To be sure, I support journals requiring source code disclosure for the same reason that experimentalists are expected not to just say the name of the chemical they used, but give the supplier, catalog no and lot number (and just like companies in ancient papers that no longer exist, code from old papers can often be un-runnable). But at the end of the day, it's only a problem if your results happen to be an artifact of the specific reagent/implementation you used. If the protocol works well with any reagent, and if the algorithm is not hard to implement properly, it's not really a big deal.

Answer 12

This might have to do with their lab policy or partners. I am a computer scientist working on problems with a medical focus, none of my clinical partners are okay with releasing code. This is a real problem with medical communities, they even asked me to wipe my computer when my contract was up so that I wouldn't have a copy of my own code. Of course like many, the real reason I suspect is that they think sharing the code puts them at a disadvantage, they simply don't want others to use their benchmark. This is just my hunch although it sounds counter-intuitive. But many people had no problem replicating my work and my experimental results - so the fact that you were able to implement the work in 2 weeks and achieve their results, I would say they did a decent work (plus you gained more insight and outdid their work). I had trouble getting the same results using my labmates' code at times.

On a personal level, I am thinking that if the authors didn't make their code publicly available, they would be hesitant to share because that requires cleaning the code etc. which might be time consuming for a single individual, which to be honest I can relate. Most people work in groups though so I would expect their code to be more modular and cleaner, or simply they could have someone work on it.

Answer 13

I appreciate the bad taste in your mouth, but please put yourself in the shoes of a reader who is interested in research as a distilled outcome of a methodology in the net of predicaments and accidents.

On a note that the implementation of their algorithm is your implementation

I'd expect yes. You ought to specify to which extent your work is original and in which way it differs/draws from any things you know. This is about doing justice to the work done by any other scholar, obnoxious to the bone as they may be. To each his own.

Furthermore, shifting the focus onto the topic, it can be useful to have in mind the (attempts to draw) distinctions between

• replication
• repetition
• reproduction

as discussed in https://plato.stanford.edu/entries/scientific-reproducibility/#ReplRepeReprScieResu borrowing from reflections in social sciences.

On a note that you reached out to them for their code and they declined

Here it is time to do justice to your own effort and working situation. If anything at all, I would state no more than something like

That source code is not available [to us] (that gentleman, personal communication)

No adjectives, no adverbs in any case. No allocation of blame, just an event, documentable if ever needed. If the code ends up to be available later, it was miscommunication or bad timing.

The reader will appreciate the efforts of advancing the state of the play on their part and yours alike.
At least, as a referee I would be happy like this. My two cents. Happy to be corrected.

Answer 14

Acknowledge the source of the algorithm, gratefully, state that the code is not based on the earlier code and do not complain that the earlier code was unavailable. As others have pointed out, they may lack the legal authority to give you the original code, and in any case the new code serves as independent confirmation of the algorithm.

Answer 15

It would be very useful for the community if you clarify that you have implemented your version of the algorithm and provide the details of parameters (if any) in the supplementary material or your experimental setting. You don't probably need to elaborate on why you implemented other algorithms, the readers can guess what has happened! It would be also very useful for the community if you could also publish your implementation so others can save time for their comparison, and obviously your paper becomes more visible as a result. I know this is quite a struggle for researchers to implement the work of others and conferences (and reviewers) are becoming more vigilant to make sure authors provide not only code but also any other info such as data and parameters for ensuring the replicability of the work. Hopefully, someday, we reach a point, where our publications serve as a justification for the key outcome of our work, which is the code we write.

Answer 16

No it is not appropriate and such statement or “accusation” should not be part of a scientific publication. None of the readers would be interested in that detail and it also does not serve the purpose of your paper.

Instead of pointing of that the authors refused to hand out their source code, you can simply mention that the source code is not available and the implementation is of your own. I understand that you felt offended by the author and wanted to retaliate. After all, you could verify their result with your implementation, so no doubt is left. Why should you then bring up an personal issue in a scientific publication, whose sole purpose is to damage the reputation of the other authors?

@Graham has already provided an excellent answer, which explains why it is necessary and desirable to have independent validation. I was in this “business” before and agree on all the points he made. My field was bioinformatics and we not only published new algorithms but also produce software based on that for the biologists to use. It is a common practice in our field that the source code is published because we all want to have users. Therefore, I did not have your problem when doing performance comparison. However, I had the other problem that @Graham pointed out. Many of the real performance improvements lie in the implementation, which is simply not mentioned in the publication. This could be intentional or unintentional. Some of the implementation details don’t sound like a scientific contribution and are left out. However, their impact could be huge. We were once puzzled by the performance of a software we wanted to compare with. Therefore we decide to analyze its source code. We then figured out the author has a very smart way of implementing a function, which saves a huge amount of time. This is a common mathematical function in our field and it seems quite clear how this function should be implemented until we saw his approach. We reimplemented his techniques in our software and saw wonders happened.

Answer 17

Have you been able to reproduce the results or not, from the description of the algorithm in they article? This is not clearly seen in your question.

If yes, then the article is complete and there is nothing to complain about.

If not, the, yes, there is something to say about just the wording should be different:

"We have repeated these experiments but were not able to reproduce the results"

In order to be a scientific publication, the article must include enough information to repeat the experiment. If multiple independent researchers fail to reproduce, the author gets into trouble.