77

I work in a sub field of computer science. There was a recent paper (2020) that was published in a reputable conference that claimed to achieve state of the art performance on a specific data set. This data set is a hot dataset, and my research group has many papers on that specific data set (improving state of the art). I asked the authors very politely if they can share their code samples for their fairly complicated algorithm, to which they rejected and told me that it is not available.

One of my colleagues took 2 weeks to code up the algorithm, and verified its performance, as well as the claimed performance. Meanwhile, we reached a new breakthrough on the dataset and achieved a new state of the art. We are currently writing a draft.

Now, I can't help but be left with a bad taste from this whole experience. I don't see why they wouldn't share their code especially if they claim a state of the art on a very important data set. Most papers have to be compared with the state of the art to some extent, and so essentially, they're slowing down all researchers in this specific field to recreate their own work prior to publishing their own research.

Now my question is, would be advisable to add a note that the implementation of THEIR algorithm is our implementation, and that we reached out to them for their code and they declined?

  • 16
    It is called replication crisis. Nothing to do with you particularly. Everyone is facing it – SSimon Apr 27 at 2:57
  • 82
    @SSimon: since the colleague was able to code the algorithm from the description in the paper and to verify its performance, this is not an example of the replication crisis. (That it is still not nice(TM) is another question IMHO) – cbeleites unhappy with SX Apr 27 at 8:45
  • 6
    @user111388: I don't say it's bad if a paper contains code. I appreciate it. But at the same time I realize that in practice it will further reduce the amount of verification of said papers. The 2nd comment is still very much in the context of the replication crises: knowing that there is a replication crisis I think it fortunate that a paper was thoroughly verified - even if that was not the original intention of OP's group. One of the main reasons behind the replication crisis is that far too few replication is done. Those 2 weeks of work need to be properly appreciated! They are far more – cbeleites unhappy with SX Apr 27 at 12:10
  • 6
    ... than something that could have been achieved by using the other group's code. You are right, though, that having the code available should not deter one from properly verifying the paper. I.e., it's even better if the code is available and 2 weeks are spent on verifying it. But: OP's group now has been doing something that we may call a blinded verification, i.e. without access to the code. Verification with the code available would already be a whole lot of an advancement over what is usual in my field, but we wouldn't consider it as independent a verification. – cbeleites unhappy with SX Apr 27 at 12:15
  • 5
    I once asked some authors of a paper for source code because their paper claimed they'd implemented a functional version of their novel algorithm. After a lot of back and forth they reluctantly shared their code and it didn't work at all. Their algorithm was sound though, and after implementing it from scratch my code worked. So it's quite possible their code has problems they'd rather not share. – par Apr 28 at 10:15

17 Answers 17

124

would be advisable to add a note that the implementation of THEIR algorithm is our implementation

Absolutely yes. This provides important context for your experimentation and as such, is valuable information for the reader.

Even better, you could make your implementation of their algorithm publicly available, so that future groups of authors will not encounter the same situation as you did (and will cite your paper).

and that we reached out to them for their code and they declined?

That is the more delicate question to ask. It would surely be legitimate to do so, as it would represent the situation accurately. But such a statement has a social implication, which is that the authors appear in a slightly negative light (perhaps rightfully so). Proceed with caution.

| improve this answer | |
  • 80
    Maybe one could phrase it in the passive voice to ease the accusatory tone: "The code could not be obtained." – henning -- reinstate Monica Apr 27 at 9:04
  • 24
    @henning--reinstateMonica and that is how things become vague and useless. Not everything needs saying, but when you want to say it, say how it is in a polite manner. I can also see it creating doubt with the reader ("couldn't they simply send an email? How hard is this") similar to how I will prioritize papers without typos because at a glance they seem like having a higher chance of not being mistaken or trivial. – Luc Apr 27 at 10:37
  • 56
    I have seen this before, especially in hot topics. The paper said something along the lines that they use their own implementation to provide a benchmark for method X, as the code was not available publicly or upon request. I think just something along the lines of "The code was not available" is not vague at all. They also made sure to point out any implementation or parameter details they had to set, and whether there is given value for those parameters in the original publication, or how they determined the parameter values to use. – penelope Apr 27 at 11:30
  • 6
    Make your implementation of their algorithm publicly available. Thus, the other group can check your implementation against their code. If you made error, which makes their algorithm look bad (things can happen), then they can notify you and highlight the issue. The reason why the other group doesn't make their code available can be plenty, Hanlons razer applies. The readers of the paper can draw their own conclusions from you re-implementing the algo of the other group. – Dohn Joe Apr 27 at 13:48
  • 4
    @Luc The passive voice is often used to avoid accusations. It is omitting a fact, but that fact is arguably not relevant while the fact the could code not be obtained is. There are many, many reasons getting the code may not be as simple as sending an email and highlighting that it was an outright refusal probably doesn't help anyone while casting the other researchers in a somewhat negative light. – TimothyAWiseman Apr 27 at 19:18
83

As a software engineer, I'll give the dissenting opinion.

Source code is not an algorithm. It's a "dusty mirror" version of something which hopefully is the algorithm they intended and which hopefully performs correctly. Software being software, and coders being human, there are many ways in which those "hopefully" parts may not be as expected. It is even possible (albeit unlikely) that the perceived improvement may come from an unintended feature of the implementation, and/or that perceived improvement may be as a result of making an implementation trade-off which has deleterious effects elsewhere.

In general science terms, the algorithm is the hypothesis you're testing; and the source code is your laboratory glassware, circuit boards, interferometer, or whatever other equipment you'd use to test the hypothesis. A general principle of science is that reproduction of results must be independent. That means you don't just get a second person to blindly run the same tests on the same glassware with the same bottles of reagents - instead an entirely separate team reproduces those results from first principles to confirm them. Independent testing of a hypothesis is what moves this from merely a hypothesis to a solid theory.

By implementing the algorithm independently, you have performed a vital scientific role in reproducing the original results. If the original authors had simply given you the source code, there would have been no reproduction and test of the algorithm. If they'd given you the source code, you might as well count photocopying the paper as "reproducing their results". :)

You would like to push ahead with the state of the art, and that's great - it shows that you're ambitious in your field. But independently verifying a hypothesis to be able to call this a "theory" is not a waste of time, and slowing down to get this groundwork in place is not something which should leave you with a bad taste if the "science" part of "computer science" is for real. Otherwise you're not doing science, you're doing hacking; which is valuable in its own way, but it isn't science.

| improve this answer | |
  • 12
    I do think it is best to share the source code to aid in a constructive research community, but I agree with you. For every well written paper it must be possible to derive an implementation that obtains the same results. If this is conceivable then the source code is not mandotary, but a nice and helpful addition. – levilime Apr 27 at 12:52
  • 22
    Retroactively, after the reproduction came out positive in this case (phew!), it's nice to have it. The bigger hassle is the case where the results cannot be reproduced, and without the original implementation it's hard or even impossible to pin down what happened. – lighthouse keeper Apr 27 at 14:58
  • 3
    @lighthousekeeper I don't disagree when that happens. (That's "when", not "if" - we're all fallible!) But unless you start your reproduction from first principles, you risk your results being biased by what you get from the other team. – Graham Apr 27 at 16:11
  • 3
    @JouniSirén Anyone who thinks source code is a complete description of anything should not describe themselves as a scientist or a coder, because it demonstrates a lack of competence in both science and coding. The fact you mention implementation differences should be a very clear pointer for why the paper needs to exist, and why the algorithm needs to be made available to be reproduced, because those implementation issues are what will kill you in corner cases. (In real-time control, that could mean literally kill, by the way.) – Graham Apr 28 at 10:06
  • 5
    @einpoklum The point is that the algorithm provides correct solutions. The program is (hopefully!) an implementation of that algorithm. If the program is doing anything which the paper has not described, then the author is (almost by definition) either working in bad faith when it comes to sharing with the community or has not written their paper correctly, and the results and paper should be rejected for publication. – Graham Apr 28 at 16:52
11

You can, and maybe you should in order to make your position clear that you think that academic codes should be shared if that is feasible. Many scientists will agree that sharing codes leads to better science because it makes it easier to replicate research, and also to find mistakes.

At the same time, not everyone feels like that, and you happen to have found someone of that opinion. To publicly out them as "backward" and "unwilling to make it easier for others to replicate their findings" may be ethically the right thing to do if you believe in open software, but it may also rub them the wrong way and create "a bad taste" in their mouths to use your phrase.

So the question is: How much do you want to insist on principle, and how much do you potentially want to alienate one of your colleagues? That question likely comes down to where you and where they are in your respective careers.

| improve this answer | |
  • 5
    I agree. I think in many circles, it's also possible that the code was not shared due to internal regulations. For example, the authors may be working at a company that allows them to publish general advances in knowledge (e.g. an improved algorithm) but more strictly controls the source code (or the data sets that were used). Publishing is not purely academic. Only 6% of conference presenters at the top two AI conferences share code. sciencemag.org/news/2018/02/… – Tom Anderson Apr 29 at 9:38
  • @TomAnderson: Yes, excellent point. – Wolfgang Bangerth Apr 29 at 15:00
6

My advise is to get the most out of the situation as it is: you did a proper replication from the description in the paper (as opposed to "just" running their code) and verified their claims. Publish this as such: this is proper scientific work.

In your own interest, I'd not mention that you did not do this replication entirely of your own free will.


TLDR:

I fully agree that it is not nice(TM) to not share code (see also below).

According to what you describe, however, this inavailability of the code has lead to further scientific advance than had the code been available to you:
You verfied their claims by implementing from scratch the algorithm described in their paper without any access to their code. It is scientifically relevant to describe this in your paper since it means that you provide the next higher level of reproduction compared to "only" running their code on your computers: you also verified that the description in the paper is sufficient to reproduce the claims.

IMHO this is particularly important since we do have a reproducibility crisis in many fields.

Sharing code helps with what would be called repeatability in my field (analytical chemistry), but reprodcution or replication of a study/paper is more than that.

Yes, code sharing is nice since it saves a whole lot of work for those who don't want to replicate, it is often the easiest way to unambiguously describe what computations were done, and is also very helpful in tracking down misconceptions by the reader. Still, iff code sharing leads to everyone (or too many) skipping the effort put into proper reproduction/replication it may be counterproductive wrt. the reproducibility crisis.

Thus, I'm a bit torn:

  • Their not sharing code is not nice.
  • Saying that you did the implementation from scratch only because the code was not available also means admitting on your side that you'd skip a more thorough verification of that paper if you have a choice. Also not nice.

The ideal situation would have been had OP's group done a full verification of the paper and have gotten access to the paper authors' code.

Personally, I'd not allude to any non-scientific reason for tackling the reproduction of that paper.

I use "not nice" here to indicate behaviour that is less than ideal (for scientific advancement) but is also clearly within the allowed range of behaviour. To be clear, I don't want to force anyone into reproducing other groups' studies. At the same time, if all groups deny to do this kind of "homework", the risk for replication crises increases.


There are levels of not sharing that I consider definitively problematic from a scientific point of view:

  • Your proper efforts cannot reproduce the claims and the authors refuse to guide you (whether by sharing code or by looking at your code and telling you what needs to be done differently)
  • They claim in the paper that they will share the code, or the journal guidelines stipulate that they'll share but they refuse.

So far, the question doesn't indicate any of these points apply.


To chime in with Anonymous Physicist's answer: I have been working at a research institute where the very official line was that neither code nor data are published with the paper ("are available on reasonable request" was the official wording) - that I would have liked to not only show my research code but put it under an open license did not matter at all.
Thus, had I been author of said paper, you'd have gotten a polite reply that I cannot give you the code without official administrative permission, and would you please contact the director about it. However, I can easily imagine institutes specifying that whoever is contacted about sharing the code has to reply in the negative without creating further work to upper management...

From this experience: if you critizise the authors for not sending the code please make sure that you attach the blame to the correct persons.

| improve this answer | |
  • 3
    "At the same time, if all groups deny to do this kind of "homework", the risk for replication crises increases." An important difference in computer science, compared to other disciplines, is that the effort for replication is minimal if the original authors provide their code and data (with proper documentation). In good conferences and journals, it's becoming increasingly common practice that the authors are required to do so. – lighthouse keeper Apr 27 at 10:55
  • 1
    @lighthousekeeper: But that's not a full replication: the effort is minimal iff they use that code without checking whether the paper contains a sufficient description of that code. Yes, there may be field-specific differences here (my field, analytical chemistry looks at repeatability, reproducibility and replicability in levels of increasing difficulty and independence. For us, it is very important to not call a replication what is only a repetition.) I fully agree with you a) that if the authors were required to share, it is a serious breach if they don't and b) that one should work... – cbeleites unhappy with SX Apr 27 at 11:02
  • 2
    ... towards requiring this even more. But unless the publication was under any such requirement, it's not nice, but not an academic misconduct and OP cannot criticize that other than as a political opinion (which is fine and laudable, but not scientific content of their paper). – cbeleites unhappy with SX Apr 27 at 11:04
6

Without knowing why the authors declined, it's a bit hard to tell if the other authors were "nice" or "naughty".

There are some legal and institutional pitfalls when it comes to sharing code. As an example, the university may have the right to check if the concrete research output contains patentable material. The authors may be forbidden from sharing their code if this check is ongoing.

So I suggest avoiding putting a negative light on this interaction if possible. Yet, it is important for the reader to know that you wrote your own implementation. And it's also important for the reviewer as well, as the reviewer may think that the other approach looks unnecessarily bad because you didn't implement it well.

Many papers have an appendix that will not be published with the rest of the paper. You could add a note for the reviewers that sharing the code with you has been refused for unknown reason, so you had to implement your own version. If you want to mention in the main part of the paper why you didn't use the original implementation of the other approach, you could mention that it was unavailable, hence you implemented your own version and made sure that you took similar care in both implementations not to miss important optimizations.

| improve this answer | |
4

I don't see why they wouldn't share their code especially if they claim a state of the art on a very important data set.

Hypothetically, one can make software that can't legally be distributed under copyright law.

If I own a book I can write notes on it and cross out sections all I want - but I can't distribute copies of my modified book. Likewise, if I want to extend a copyrighted Matlab library in a way the authors didn't anticipate, I can copy-and-paste from it and modify it as much as I like for my own use - but I can't distribute the resulting derivative work.

If you want to assume good faith, you could assume some similar benign reason on their part.

Of course, a cynic would say authors are rewarded for wringing as many papers as possible out of a piece of work, and if there are obvious follow-on papers (like obvious tweaks, or tests on other datasets) not releasing the code until you've written those papers is the behaviour their paymasters reward.

| improve this answer | |
  • 1
    It is quite easy to make a software that cannot be legally redistributed: link together open source libraries which have incompatible licenses. See opensource.org/licenses; it is widely known that GPLv2 is incompatible with GPLv3 – Basile Starynkevitch Apr 30 at 4:37
  • This answer rightly addresses a minor question contained in the OP. However, the main question from the last line is "would be advisable to add a note that the implementation of THEIR algorithm is our implementation, and that we reached out to them for their code and they declined?". It would be helpful if you could address also this last point. – Massimo Ortolano May 9 at 9:10
4

With the information given, it is not possible to assess the motivation of the original authors. As a frequent publisher of open source scientific code, I can think of several.

  1. They did not want anyone to replicate their results using their code, as they wanted to make more publications from it (certainly not nice, but seen before).
  2. Their code was not in a state to be used be anyone else (PhD-ware), and they don't have the man-power to clean it up.
  3. They make use of proprietary libraries, or have internal licensing issues, not allowing them to share the code.
  4. They simply don't want to be bothered by the inevitable tech support issues, that comes with supplying ones code to other people.

And the list could be continued.

You can only act given the information at hand, and that is, that at the moment the code does not exist in an open form. Fortunately, you were able to reproduce the results, and you are now free to publish your own implementation along with your article, prompting users to cite your paper, as well as the original one. After all, you have made a significant piece of scientific work, which should now be credited.

You don't need to be snarky in your paper. Write about what you did, and not what other people did not. You can note that:

The implementation outlined in this paper reproduces the performance by author A, quoted in ref. [A]. The present implementation can be obtained from XX, allowing for further independent validation.

| improve this answer | |
2

Unless the topic of your research is open source software, your journal article is not the right place to criticise your colleagues for not releasing their source code. Journal articles and conference papers should be about the knowledge you have created, not your opinion of your colleagues.

Before criticising your colleagues in any venue, you should consider if releasing the source is a reasonable option for them. There may be legal restrictions you do not know about, or a very inconvenient administrative process.

Peer review is a great time to encourage transparency.

| improve this answer | |
  • 13
    A statement like "Despite our request, the original implementation of the algorithm was not made available to us." is not an opinion, but a fact, and might be relevant to explain why it was necessary to re-implement the algorithm (possibly with different results). People might be disgruntled anyways. – lighthouse keeper Apr 27 at 7:57
  • 4
    @lighthousekeeper Your fact is still criticism, and clearly implies an opinion. I would suggest "We reimplemented the algorithm" as a more appropriate wording. The request is irrelevant. – Anonymous Physicist Apr 27 at 9:17
  • 11
    "We reimplemented the algorithm" might raise the question why a reimplementation was done. OP seems to be in the comfortable situation that both implementations produce the same results. If the results would have been different, one would need to justify what one did to figure out the cause of the differences. If the original implementation was not available, one would need to state precisely this. The request would be relevant, because it signifies more initiative on the authors' side than just saying "it wasn't available". – lighthouse keeper Apr 27 at 9:25
  • 4
    Personally, as far as I am concerned, if your code cannot be realeased for legal purposes, then your paper cannot be published. I think a passive statement that just states the facts "We reimplemented the algorithm as the orginal code could not be obtained", as suggested by @henning above is the way to go. – Ian Sudbery Apr 27 at 9:40
  • 6
    @AnonymousPhysicist That might be a field-specific difference then. In computer science, rationale is an important part of scientific writing. OP mentions that they "verified the claimed performance". Apparently they came up with a different algorithm that yields better performance than the reimplemented one, so it's key that the performance of the original algorithm was properly reported. – lighthouse keeper Apr 27 at 10:47
1

Is it appropriate to mention that authors refused to provide code samples when asked?

I work in a sub field of computer science. There was a recent paper (2020) that was published in a reputable conference that claimed to achieve state of the art performance on a specific data set.

There are several issues.

FWIW, I am a semi-academic research engineer in France, working at CEA LIST in a cybersecurity lab, where I develop Bismon (a GPLv3+ software funded by two H2020 projects: CHARIOT and DECODER and described in this draft report)

  • how was the work described in the paper funded? if the funding agency is military (e.g. DARPA) the funding contracts may require academic publications but disallow any publication of the code or of the data set. Even H2020 funding may forbid publishing of data, or simply getting permission to publish the data or the code may require a lot of red tape work that usually is boring for any academics.

  • who owns that dataset and how was it funded? If the dataset was difficult to collect (think of biological data, e.g. Covid19 related, or of nuclear industry related data) you might not be able to access it

  • are there any privacy issues or personal data issues? Regulations like GDPR are then highly relevant

  • what legal system and copyright applies to both the data and the code samples? Read also Jean Tirole's work, in particular his paper with Josh Lerner on simple economics of open source. Notice that Shoshana Zuboff books and papers (e.g. Big Other: Surveillance Capitalism and the Prospects of an Information Civilization for example) are also relevant.

  • the dataset or the code may be an asset for future research grants.

Notice that academia is a small world (read also about Ershov number and more importantly Erdös number). Mentioning in a formal written paper that some authors refused to provide code samples may backfire, and could perhaps impact negatively your academic career in the next few years. The common practice is dual: mention in acknowledgment sections the people and their institution who provided you with code and data.

If publishing the code or the dataset was not previously and explicitly mentioned in the research grants funding the work, it is likely that you might not get it easily. I got the habit of mentioning (at least in footnotes) such things in most research grant proposals I contributed writing.

By personal experience, obtaining permission -within professional context- to publish some code as open source or some dataset as open data is very boring and stressful (and nearly impossible after the work has started). You should mention these things in writing in research proposals and grants.

there is no such thing as a free lunch

See also these slides, and the RefPerSys project. So far, funding is the main problem in it.

In Europe, ITEA3 and BDVA and HorizonEurope could give interesting contacts. You should find the equivalent in your place.

| improve this answer | |
  • 1
    Ershov number? Or Erdos number? – avid Apr 29 at 9:49
  • One would hope that authors would write into the paper "this dataset is not avaible because of miltary funding".. – user111388 Apr 29 at 9:57
  • 1
    No, writing that requires approval by military, and getting such an approval require days of full time work – Basile Starynkevitch Apr 29 at 9:57
  • So what? Then the authors should consider it part of their work to improve that paper by getting the approval for this sentence. What is the point of this rule (one is not allowed to write in a paper for which reasons the data is not avaiable) anyway? – user111388 Jul 24 at 20:48
0

I appreciate the bad taste in your mouth, but please put yourself in the shoes of a reader who is interested in research as a distilled outcome of a methodology in the net of predicaments and accidents.

On a note that the implementation of their algorithm is your implementation

I'd expect yes. You ought to specify to which extent your work is original and in which way it differs/draws from any things you know. This is about doing justice to the work done by any other scholar, obnoxious to the bone as they may be. To each his own.

Furthermore, shifting the focus onto the topic, it can be useful to have in mind the (attempts to draw) distinctions between

  • replication
  • repetition
  • reproduction

as discussed in https://plato.stanford.edu/entries/scientific-reproducibility/#ReplRepeReprScieResu borrowing from reflections in social sciences.

On a note that you reached out to them for their code and they declined

Here it is time to do justice to your own effort and working situation. If anything at all, I would state no more than something like

That source code is not available [to us] (that gentleman, personal communication)

No adjectives, no adverbs in any case. No allocation of blame, just an event, documentable if ever needed. If the code ends up to be available later, it was miscommunication or bad timing.

The reader will appreciate the efforts of advancing the state of the play on their part and yours alike.
At least, as a referee I would be happy like this. My two cents. Happy to be corrected.

| improve this answer | |
0

Some ramblings -just to provide another perspective- before the answer.

I can share my experience as a "user" for similar occasions. CS is not my field, and I can not rewrite each and every algorithm that claims to be a "drastic improvement" (Spoiler alert: Most are not). I simply do not have time, and brute force algorithms usually work for me. Usually. But if I had access to said algorithms (read: actual code), I would have used them and that would be a huge enabler for my future research.

Some answers/comments claim, roughly stating, using the same code is not reproduction. For CS, this might be true to some extent -I won't even argue that. But reproduction is the real concern here? I value usability more than pure algorithm reproduction.

There are other fields (such as mine, sports sciences) which would use some "help". Don't take it wrong way, I can most probably code said state of the art code myself, given the algorithm. But it simply is not worth my time. Because a potential (and a very little one in my past experiences) improvement in execution speed/memory footprint does not justify working more on my already-working proof-of-concept code than tackling the next problem on my list.

There are lots of articles proposing various algorithms, but not bother to even explain them properly.

Back to your question:

Now my question is, would be advisable to add a note that the implementation of THEIR algorithm is our implementation, and that we reached out to them for their code and they declined?

Definitely advisable for the first part, since that is pure blind reproduction of their algorithm, and it adds value to both parties' research.**

For their refusal: How can you be sure that their proposed algorithm was not a product of a lunch-time conversation, and never before realized in actual code until you did it? How can you be sure their claimed performance figures were not fabricated? You simply trust the researchers? What if your implementation did not verify their algorithm's superiority? Would you still publish it, or give in to appeal to authority? Would you check -rigoriously- if their code match up with their explanation of the algorithm: i.e. there is nothing up their sleeves?

I believe these questions are the reason their refusal leaves a bad taste; not your coworker's 2 weeks.

Whether in a publication/presentation or in your circle, I think you owe science to make this info public. The latter seems less risky if you are not "well established yet" though.

| improve this answer | |
0

The whole providing source code of the paper thing has been discussed to death on this site and elsewhere, so I won't get into it much. I agree with you that they should have given source, and it should be required by journals anyhow. That said, currently it is not. It would be nice if they helped you reproduce their work, but they don't have to.

If you had discovered that when you take their instructions at face value the results are not reproducible, you would have cause to write to the editor of the journal and the onus would be on them to help demonstrate their own findings as legitimate. However, it sounds like you[r friend] were able to reproduce it without their help, you just wish it was easier. Fair enough, but you can hardly demand they go out of their way to make your life easier.

If you intend to distribute the code you wrote, there is nothing wrong with that and it's a very good idea. I wouldn't attach a little manifesto saying the authors were unhelpful jerks and you wrote this prove your point. While it is true and you are within your rights, it will make you look bad. It's worth explaining that your implementation is independent and not based on the original code, so as to not confuse any readers, but there is no need to go into why they didn't share or who is at fault -- these are not matters of fact pertaining to the research itself. You can say something like "this code is an independent implementation of the algorithm in Smith et al. 2020, written without access to the authors' own implementation". That last bit you can fine tune depending on how fine a point you feel like putting on it. But, once again the less you say about how much you dislike that the authors wouldn't share the code with you, the better (in the long run).

I don't see why they wouldn't share their code especially if they claim a state of the art on a very important data set. Most papers have to be compared with the state of the art to some extent, and so essentially, they're slowing down all researchers in this specific field to recreate their own work prior to publishing their own research.

There can be many reasons, not all of them nefarious.

  • If the journal doesn't require code, providing it is extra work and inefficient.
  • You would have to document how other people can build and run it on their own computer - something which can take extra work and maybe is not a concern on your own system.
  • Generally the code is not that helpful if not portable - while the algorithm is already portable.
  • Just because they provide the code doesn't mean it will be useful to you. It could be in a language you don't know.
  • The code may be messy and perhaps they don't feel like inviting pointless questions. This may seem dubious but imagine you were asked to provide every single draft and napkin drawing, and then people made you explain random thoughts that were jotted in the margin and were abandoned later on anyhow.
  • Whenever you provide code, all sorts of people start asking you for help with matters that are sometimes barely connected to the work itself. While you should be available to answer questions about your work, answering questions takes time. If the journal didn't require it, why create additional work for yourself?

To be sure, I support journals requiring source code disclosure for the same reason that experimentalists are expected not to just say the name of the chemical they used, but give the supplier, catalog no and lot number (and just like companies in ancient papers that no longer exist, code from old papers can often be un-runnable). But at the end of the day, it's only a problem if your results happen to be an artifact of the specific reagent/implementation you used. If the protocol works well with any reagent, and if the algorithm is not hard to implement properly, it's not really a big deal.

| improve this answer | |
0

Acknowledge the source of the algorithm, gratefully, state that the code is not based on the earlier code and do not complain that the earlier code was unavailable. As others have pointed out, they may lack the legal authority to give you the original code, and in any case the new code serves as independent confirmation of the algorithm.

| improve this answer | |
0

It would be very useful for the community if you clarify that you have implemented your version of the algorithm and provide the details of parameters (if any) in the supplementary material or your experimental setting. You don't probably need to elaborate on why you implemented other algorithms, the readers can guess what has happened! It would be also very useful for the community if you could also publish your implementation so others can save time for their comparison, and obviously your paper becomes more visible as a result. I know this is quite a struggle for researchers to implement the work of others and conferences (and reviewers) are becoming more vigilant to make sure authors provide not only code but also any other info such as data and parameters for ensuring the replicability of the work. Hopefully, someday, we reach a point, where our publications serve as a justification for the key outcome of our work, which is the code we write.

| improve this answer | |
0

This might have to do with their lab policy or partners. I am a computer scientist working on problems with a medical focus, none of my clinical partners are okay with releasing code. This is a real problem with medical communities, they even asked me to wipe my computer when my contract was up so that I wouldn't have a copy of my own code. Of course like many, the real reason I suspect is that they think sharing the code puts them at a disadvantage, they simply don't want others to use their benchmark. This is just my hunch although it sounds counter-intuitive. But many people had no problem replicating my work and my experimental results - so the fact that you were able to implement the work in 2 weeks and achieve their results, I would say they did a decent work (plus you gained more insight and outdid their work). I had trouble getting the same results using my labmates' code at times.

On a personal level, I am thinking that if the authors didn't make their code publicly available, they would be hesitant to share because that requires cleaning the code etc. which might be time consuming for a single individual, which to be honest I can relate. Most people work in groups though so I would expect their code to be more modular and cleaner, or simply they could have someone work on it.

| improve this answer | |
-1

No it is not appropriate and such statement or “accusation” should not be part of a scientific publication. None of the readers would be interested in that detail and it also does not serve the purpose of your paper.

Instead of pointing of that the authors refused to hand out their source code, you can simply mention that the source code is not available and the implementation is of your own. I understand that you felt offended by the author and wanted to retaliate. After all, you could verify their result with your implementation, so no doubt is left. Why should you then bring up an personal issue in a scientific publication, whose sole purpose is to damage the reputation of the other authors?

@Graham has already provided an excellent answer, which explains why it is necessary and desirable to have independent validation. I was in this “business” before and agree on all the points he made. My field was bioinformatics and we not only published new algorithms but also produce software based on that for the biologists to use. It is a common practice in our field that the source code is published because we all want to have users. Therefore, I did not have your problem when doing performance comparison. However, I had the other problem that @Graham pointed out. Many of the real performance improvements lie in the implementation, which is simply not mentioned in the publication. This could be intentional or unintentional. Some of the implementation details don’t sound like a scientific contribution and are left out. However, their impact could be huge. We were once puzzled by the performance of a software we wanted to compare with. Therefore we decide to analyze its source code. We then figured out the author has a very smart way of implementing a function, which saves a huge amount of time. This is a common mathematical function in our field and it seems quite clear how this function should be implemented until we saw his approach. We reimplemented his techniques in our software and saw wonders happened.

| improve this answer | |
-2

Have you been able to reproduce the results or not, from the description of the algorithm in they article? This is not clearly seen in your question.

If yes, then the article is complete and there is nothing to complain about.

If not, the, yes, there is something to say about just the wording should be different:

"We have repeated these experiments but were not able to reproduce the results"

In order to be a scientific publication, the article must include enough information to repeat the experiment. If multiple independent researchers fail to reproduce, the author gets into trouble.

| improve this answer | |
  • I have rewritten this question and think the current version is correct. – h22 Apr 28 at 10:25

Not the answer you're looking for? Browse other questions tagged or ask your own question.