64

I have been working with my research group on a number of papers in a certain area of Computer Science. These papers have a certain portion which is theoretical (i.e. contains theorems, formalisms, algorithms) and a certain portion which is experimental (i.e. describes implementations and empirical results).

With regard to the experimental/practical sections, my supervisor generally suggests prioritizing the information that illuminates the general thrust of the paper (i.e. the section in the paper itself along with figures, etc) over the actual implementation details like links to code. Even if he considers releasing some tangible material, he seems to prefer to release executables rather than source code.

I can understand the rationale behind this because:

(1) In Academia, the final paper is the most important and recognized artifact of research;

and

(2) Releasing unpolished source code could be embarrassing, because it might contain errors which could damage the reputation of our research group.

While I agree with this, I am conflicted over what this means for the value of the paper and the ethics of research.

  • The papers do have some theoretical content, but they do not seem to be particularly valuable. Anyone else could have come up with those ideas with a little bit of thought.

  • If there are no executable artifacts tied to my paper, I could just have been lying about my results. I am not, and the experiments are actually very rigorous, but this is fact is somewhat undermined

  • Considering that a large part of our experimental work depends on comparing with artifacts created by other research groups, it seems petty not to release our artifacts

  • The potential benefit of the transparency of having open source code seems to outweigh the potential harm from potential embarrassment that may result from bugs found by the community. In my experience, artifacts released by other research teams in similar areas do often contain bugs, but I still think positively of them.

Please help me understand if my argument has any merit, or if I am being irrational.

6
  • 43
    "Releasing unpolished source code could be embarrassing" — a lot of science code is more script-like and unpolished. Would you judge some other team based on their code or be grateful that they released the code at all and understand that we all have circumstances? "embarrassing, because it might contain errors" — would you prefer to be wrong and non-embarrassed? Or to know the truth and have the chance to get your paper right? If someone finds an error in my code, I'm not embarrassed. I'm proud that someone found my code important enough to read it throroughly. And I'm grateful to them.
    – Džuris
    Jun 21 at 3:35
  • 15
    What good is a physics paper if every paper isn't also shipped with the unique equipment necessary to reproduce the results? Or a biology paper without samples of what's being studied? It's not a perfect analogy, and one can certainly make a good case that one should always release the source code, but lack of released code doesn't make a paper useless.
    – NotThatGuy
    Jun 21 at 11:06
  • 9
    @NotThatGuy that seems very focussed on the title - the text of the question is clear about relative value and the pros and cons of releasing the code.
    – mjaggard
    Jun 21 at 13:18
  • 2
    Won't you be just as embarrassed if the output of the released executable contains errors? With source code available, others may be able to track down the issues to report and or correct the code.
    – doneal24
    Jun 21 at 15:51
  • 8
    Your code has errors. My code has errors. Everyone's code has errors. If I can look at the source, I will be more convinced that you don't have too many errors. And if you hide it instead, personally, I think you know you have more errors than you should. Also, source can be re-built in the future; a compiled binary may depend on some runtimes that become hard to find in the future.
    – Davidmh
    Jun 21 at 22:08

8 Answers 8

86

I think you have this all correct. The world would be a better place if all papers released all of the software that is used to generate the results shown in it. And that's not just a personal thought of mine (and apparently of yours as well), it is empirically verifiable: Papers that release the software used get more citations than papers that do not -- in other words, others are also thinking that that is worthwhile.

People have all sorts of reasons not to release their software, including (i) they believe that they have a competitive advantage by keeping their software to themselves, (ii) they do not trust their own software, (iii) they do not comment their own software or otherwise use good software engineering practices, and don't want the world to see so, (iv) they do not want to provide support to others who would download the software and use it. I believe that (iv) is a legitimate reason. (i) is misguided in my opinion, because at least if the software is non-trivial, others trying to use it more likely than not will ask to collaborate with the authors of the software, rather than just use it themselves; in other words, the original authors would gain a competitive advantage rather than disadvantage from releasing the software. (On this point, I speak from many years of experience.) Finally, if someone does not want to release their software for reasons (ii) or (iii), I believe this is an ethically questionable approach: If you have no confidence in your work, you probably shouldn't publish it.

In any case, I believe that your arguments are correct. Whether they convince your adviser is, of course, a different question.

13
  • 11
    To your last comment point (iii); one can justifiably have confidence in their work being correct, even if it does not conform to good software engineering practices. Other than that, good answer. Jun 20 at 10:23
  • 7
    @Grzegorczyk I agree, but your output affects your livelihood in many systems of researchers' job evaluations, and so I'm tolerant of people wanting to have a competitive advantage. Jun 20 at 12:06
  • 14
    @Thissitehasbecomeadump. Empirically, the software design and development community has found that code that is poorly written and poorly documented has a much higher density of bugs. If you know enough about software design to be embarrassed by your code, you should probably be also concerned about its correctness. Jun 20 at 12:08
  • 11
    It's worth noting that well-regarded conferences (AAAI and ISWC come to mind) now have a "reproducibility" peer-review criterion, where reviewers are asked to assess the reproducibility of the work and the quality of provided supplementary materials. Just a few years ago, I didn't see this sort of thing. So the community is taking steps to push people to release code, data, etc. which supports the views expressed here (in both question and answer). Jun 20 at 13:11
  • 5
    @preferred_anon Maybe both. If you put code on the web for others to download, then others will ask you questions about it, will ask you for help to implement extensions, etc. They will, in other words, ask for your time. You may or may not be willing to offer that time. Jun 21 at 0:38
13

Check out this famous paper (in AI) by Google. It described an algorithm by which an AI can learn to play a game - in this case chess and shogi - by playing against itself only, and how such an AI can beat the then-strongest engines at the games.

Google didn't release the source code. But! The papers contain enough information for someone else to duplicate their work. Using the same methodology, the chess engine community created Leela Chess Zero, and tuned it (Leela is likely stronger than AlphaZero at this point). The ripples of the new AI continue to be felt, since 1) it affected how Stockfish, the strongest traditional chess engine, was developed; Stockfish is likely stronger than Leela again, and 2) the gameplay ideas that Leela discovered (and continue to discover) have been incorporated into the repertoire the top human grandmasters.

Now we can ask the question: what good is this paper if the code is not open source? How do we know Google didn't fake their results? I'm sure you can see the answers in the second paragraph above.

8
  • 40
    I think this answer misses the point, though. You are considering a sample of one, written by a highly regarded collaboration, writing a paper for which it would have been clear to anyone who saw it as a draft that it will be a high-impact one. The question is whether you can generalize from this paper to recommendations for everyone -- and I don't think you can. Just because that one paper had enough information to be useful to others that they could reverse-engineer things doesn't mean that -- in general -- papers shouldn't bother with including their source code. Jun 20 at 14:18
  • 3
    @WolfgangBangerth Just because that one paper had enough information to be useful to others that they could reverse-engineer things doesn't mean that -- in general -- papers shouldn't bother with including their source code. The question isn't "should papers include their source code?" however. It asks about the value of the paper given the source code isn't available.
    – Allure
    Jun 20 at 23:41
  • 7
    @WolfgangBangerth I feel like we're not reading the question the same way. From my perspective, the question doesn't ask if papers with code have a greater value than those without, either. It only asks if the paper can have value if the code is not available.
    – Allure
    Jun 21 at 1:47
  • 4
    @Allure Ah, fair enough, thanks for pointing out the disconnect. I guess the answer to that is "yes", as that's how we've been operating for a long time and sciences has progressed with that mantra. Though I continue to think that a sample of one is not the way to make that point :-) Jun 21 at 1:58
  • 5
    @Mazura You might want to ask RedHat/SUSE, Oracle MySQL, Oracle Solaris, IBM Websphere, MongoDB, Docker, Chef, Puppet, etc. on how to succeed commercially with open source code.
    – doneal24
    Jun 21 at 15:59
13

While I won’t comment on your particular case, I think it’s useful to consider science as a whole. You should realize that computer scientists are uniquely able to distribute an entire exact experiment implementation broadly, instantly, to anywhere in the world by virtue of making the source code public. Anyone can reproduce such an experiment exactly, and this is (in principle) a benefit. However, for any physical or biological experiment, there is no way to transport the apparatus to everyone. The journal text, figures, published data, etc. are all anyone can hope to have to learn of an experiment and judge its conclusions and correctness. There is an implicit assumption (barring evidence otherwise) that the researchers competently performed the experiment and collected the data they describe. Trust is a fundamental component of scientific communication, and this is why malfeasance which abuses this trust (such as data fabrication) is taken so seriously.

All of this is to say that every other experimental science views journal articles, with all their flaws, as having value; they have been the primary vector for dissemination of cutting-edge results for well over a century. They work because there is generally a strong culture of scientific integrity among scientists, and people for the most part take the presented data at face value even though the experimenters could be incompetent or untrustworthy. So although you enumerate reasons why journal articles are to be mistrusted, they have served science through the period of the greatest expansion of knowledge in human history. And they shall serve you just as well.

19
  • 2
    Your first sentence might be a little toooo optimistic: the precise platform, network situation, and competing loads on the machine(s) running something can have significant effects. Jun 21 at 3:05
  • 2
    I was going to say the same as @paulgarrett: Have you ever tried to provide a replicable environment for anything other than a 200-line Python script? Have you ever tried to actually reproduce a computational experiment? What do you do if that was run on a 10,000 core cluster? What if the software uses a dialect of some language that is no longer easily supported by modern compilers? Jun 21 at 3:39
  • 4
    @WolfgangBangerth I’m no computer scientist, so perhaps I have a too romantic notion of it’s work. I imagine that at least measurements of relative performance comparisons would be possible to replicate nearly if not exactly. But the more wrong my offending statement is, the truer the rest of my answer: Journal articles have value, as evidenced by the last century+ of experimental science as a whole, wherein experiments are impossible to replicate exactly, and so the community must trust the written word.
    – Gilbert
    Jun 21 at 5:02
  • 1
    @Gilbert Having worked in engineering/physics research, i can state that experimental details in papers are often quite flocculent and a lot of emphasis is given to insignificant things to no more avail than showing off the author's knowledge. What's more, vital experimental details are frequently omitted on the unspoken hauteur that these should be well-known to researchers of the field. Sometimes things observed by some are never observable to many others: it's just a folly no one wants to contradict. Computing may not be as variable but implementation details plus inputs are important.
    – Trunk
    Jun 21 at 11:19
  • 3
    @Trunk yes, I agree with your assessment of experimental papers in engineering/physics research. My point is that despite all of the obvious flaws, journal articles are clearly regarded as valuable even for fields for which anything analogous to publishing source code is impossible.
    – Gilbert
    Jun 21 at 11:56
9

Going point by point through your issues...

The papers do have some theoretical content, but they do not seem to be particularly valuable. Anyone else could have come up with those ideas with a little bit of thought.

This is undervaluing your contribution. Your paper, by definition, does have some useful theoretical basis otherwise you wouldn't be publishing it. It may seem obvious to you, but that's likely only because you've been living with this for a while. And I'll remind you that Thomas Henry Huxley's response on first reading Origin of Species was "How extremely stupid not to have thought of that!" What's apparently obvious after you know it may not be obvious before.

If there are no executable artifacts tied to my paper, I could just have been lying about my results. I am not, and the experiments are actually very rigorous, but this is fact is somewhat undermined

And this is why it's important for other places to independently check results. It's not sexy, but it catches dodgy results produced through incompetence or malice.

Considering that a large part of our experimental work depends on comparing with artifacts created by other research groups, it seems petty not to release our artifacts

And if they've got the same dataset as you and the same code, they'd get the same results if they just ran your code. The important thing isn't the code, it's the algorithm. Which is NOT the same thing.

The potential benefit of the transparency of having open source code seems to outweigh the potential harm from potential embarrassment that may result from bugs found by the community. In my experience, artifacts released by other research teams in similar areas do often contain bugs, but I still think positively of them.

But what if the "interesting" feature of your results is actually a coding bug? Someone simply blindly copying your code will get the same results. Independently coding the algorithm though is unlikely to get the same bug twice, so they can report that they don't get the same results and your paper may be incorrect. Having the source code would let them analyse why your results were dodgy, sure, but that's less important than discovering the fact of it being dodgy. And more likely, they'll be in contact with you (or your supervisor) after they discover they can't reproduce it, and then you'll be checking your own source to find where it went wrong and publishing a correction.

5
  • 1
    Having the source code would let them analyse why your results were dodgy, sure, but that's less important than discovering the fact of it being dodgy. But how may they discover it to be dodgy - assuming that the algorithm was verifiable - if they don't do an implementation themselves or use OP's implementation ?
    – Trunk
    Jun 21 at 11:26
  • @Trunk I don't understand your question. It is only possible to reproduce results by carrying out the process yourself. That's the definition, the only definition ever, of "reproducing". At which point they've done an implementation themselves, right?
    – Graham
    Jun 21 at 12:13
  • Or looked at OP's source and debugged/refined it. But I'm not too clear on your viewpoint here. At one point you are concerned about an algorithm error. Later you are concerned that paper readers using OP's source with or without his dataset will only generate the same results as OP. Are you concerned about the possibility of under-, over- or mis-specification in the algorithm ? Or are you suggesting that paper readers are better off doing their own implementations to reduce implementation errors and bugs ?
    – Trunk
    Jun 21 at 12:43
  • 3
    @Trunk Re the algorithm error, it's possible (I've seen it myself) that an feature in the results gets misinterpreted as supporting the algorithm as written, when in fact it turns out to just be a feature of the implementation. If you were to implement the algorithm as written then it doesn't actually do what the paper says it does. :) Whether this is a bug in the implementation or just bad writing-up in the paper, either way it's something that can only be spotted with a clean implementation.
    – Graham
    Jun 21 at 14:42
  • I see. So you are in the publish-the-source-code camp. But anyone with a mind to implement this algorithm will ipso facto have a bias towards or against a successful outcome. Maybe OP could ask a coding buddy to do an implementation schema for him. As regards the algo, maybe someone into formal verification in his department might lend a hand.
    – Trunk
    Jun 21 at 15:16
8

The goal of a paper is to report a novel research finding. It can accomplish this perfectly well without providing source code.

It is not possible to describe empirical findings in full detail in a scientific paper (or arguably any format). It ultimately falls on the author to decide what detail is appropriate.

The scientific method is not about believing observations because they have been recorded in excellent detail, but about independently verifying them. So it doesn't particularly matter whether you provide the implementation code. What matters is whether it's clear to other researchers how they would implement their own version such that it confirms the results you obtained. Your code may or may not be an essential aid for that.

Some, including me, would argue that the virtue of a scientific finding is simplicity. In your paper, the more quirky things there are that have to be done "just right" - whether the algorithm works when implemented in a certain way with a certain language, or it only works on a certain kind of input - these all make the finding itself less interesting to begin with. So if your paper is really good enough to care about, the code is a moot point. I should be able to write my own code from scratch and verify your algorithms and theorems.

However, beyond the primary goal of justifying a claim, there are secondary goals that are well-served by making the implementation available:

  • If your implementation has a mistake, it will be found sooner
  • If your implementation turns out to be correct but hard to reproduce, you will not be accused of fabricating results
  • Researchers wishing to extend on your work will have a better starting point
  • Non-researchers (such as those in the industry) wishing to apply your findings will have a better starting point
  • People trying to learn about research in the field will have better resources
  • The code will serve as a public demonstration of your coding ability, if required by for example job applications

I personally would prefer if code was always published along with papers, but there have been many papers in CS and other fields that were published without code, and yet their claims are sufficiently credible and the findings they report are useful, so I would not consider it a hard requirement.

But to your concern about the code being unpolished: It doesn't really matter. No code is perfect. Either what you have works which means it's good enough, or it doesn't work which means your paper is wrong in the first place. You have little to lose from publishing code.

You also mention publishing binary executables, which is the really strange part to me. By publishing executables, you are asking people to trust that you did not introduce any malicious code (like viruses) into the binary, and also that your system and every other system in the chain (like the CDN actually hosting your binary) is secure from hackers and so forth. This is a very big claim, and completely unnecessary when you could just provide source code and completely obviate it. You are also training users to accept bad security practices. People who do publish binary code (eg. proprietary vendors) will at least take steps to mitigate the security hole, by employing security engineers and signing their binary builds. If your goal is to let people use your implementation without seeing the code, you should provide a web server so that the binary is executed on your system, or at least containerize it and provide something like a Docker image.

1
  • 1
    A comprehensive and well-balanced answer. The fourth paragraph is beautiful. However I can't quite agree with “the code being unpolished: It doesn't really matter”. In many cases, just spending one day to go over the code again, structuring it to be more reliable, putting it together with the right tests, and adding some comments, could save people trying to use it multiple days of work to get it to run, or even weeks of needing to re-do computations because they used the code wrong. Jun 22 at 12:46
2

Unpublished papers, in and of themselves, are almost entirely worthless, like an unfilled lottery ticket, or a movie script that's never been filmed.

Once published, they get a little value, like a lottery ticket that has had the numbers filled in and registered at the checkout, or a movie that has been shot. They can all now be tested, and have a chance to win.

The community ascribes them value at this point, but this is equivalent to the face value of the scratchcard (a dollar, perhaps): there is a chance that the paper might turn out to be a "winner", based on correct data and correctly applied algorithms. Every panel on a scratchcard could be a winning number... or none of them may be. A movie could be a blockbuster... or a flop.

Reproducing the results is the proving ground: it's how they provide value to the consumer, like when lottery numbers are drawn, or a movie is released to the public.

The higher you make the bar of reproduction, the lower the chances of making any valuable contribution to the community.

The lower the bar, the more likely someone reading the journal you publish in is to say "hey, this is something I can give to my student to try" or "that's so easy to reproduce, I could try that out this lunchtime..."

There are those who argue "if you release the source, then people will run it and necessarily must reproduce your errors". This is very twisted logic: releasing the source saves them your entire development-cycle of time, meaning more people can work on what your paper gave. More eyes on your source can only mean more likelihood of finding errors: it is impossible for it to mean less.

There are cases (like the Google AI paper mentioned in another answer) where papers that involved software have been reproduced without source code, and managed to be beneficial anyway... but these are far less common. Even in the Google case, there was only one project which went through the trouble of rewriting that code. If they hadn't got lucky, if that project hadn't picked it up and run with it... what value would that paper have had to the community, then? How much MORE value would it have had if instead, they had released the source, data, and a docker file, so countless people could have just run one command and reproduced their results? How much bigger would the field of AI be now? Would we all be driving self-flying cars? We can't tell, but we can guess that the impact of the paper would not have been at all reduced by improving the reproducibility.

Assuming someone DOES reproduce your work, but gets different results... where is their error, if any? Even if you collaborate, and both use the same data to get different results... is it their error, or yours? How do they find out? What if you are not contactable? In software engineering it is considered a truism that you cannot easily identify or fix an error without reproducing it first, and if your output cannot be reproduced, it cannot be fixed.

TL;DR: All papers have some value, but without reproducibility, that value is minimal and aspirational, rather than real. Maximizing reproducibility maximizes real value and impact. Source code really helps these goals.

1

In Natural Sciences we should put forward boldly our hypotheses and results inviting debate, correction and, ultimately, falsification. Do publish your code. Why not clean it up, while are you at it? If for no one else then for yourself, who is the one most likely to re-use the code in the future.

1

The potential benefit of the transparency of having open source code seems to outweigh the potential harm from potential embarrassment that may result from bugs found by the community. In my experience, artifacts released by other research teams in similar areas do often contain bugs, but I still think positively of them.

Let me answer from a software developer's point of view. Open sourcing the code has a huge benefits, even if it contains bugs or poorly written code. Point of being opensource is other people can contribute towards it. The code will be improved by others, bugs will be fixed and will be much more popular compared to a closed source one.

So, if you are confident on results, why hesitate?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .