What good is my paper if the code is not open source?

Question

I have been working with my research group on a number of papers in a certain area of Computer Science. These papers have a certain portion which is theoretical (i.e. contains theorems, formalisms, algorithms) and a certain portion which is experimental (i.e. describes implementations and empirical results).

With regard to the experimental/practical sections, my supervisor generally suggests prioritizing the information that illuminates the general thrust of the paper (i.e. the section in the paper itself along with figures, etc) over the actual implementation details like links to code. Even if he considers releasing some tangible material, he seems to prefer to release executables rather than source code.

I can understand the rationale behind this because:

(1) In Academia, the final paper is the most important and recognized artifact of research;

and

(2) Releasing unpolished source code could be embarrassing, because it might contain errors which could damage the reputation of our research group.

While I agree with this, I am conflicted over what this means for the value of the paper and the ethics of research.

• The papers do have some theoretical content, but they do not seem to be particularly valuable. Anyone else could have come up with those ideas with a little bit of thought.

• If there are no executable artifacts tied to my paper, I could just have been lying about my results. I am not, and the experiments are actually very rigorous, but this is fact is somewhat undermined

• Considering that a large part of our experimental work depends on comparing with artifacts created by other research groups, it seems petty not to release our artifacts

• The potential benefit of the transparency of having open source code seems to outweigh the potential harm from potential embarrassment that may result from bugs found by the community. In my experience, artifacts released by other research teams in similar areas do often contain bugs, but I still think positively of them.

Please help me understand if my argument has any merit, or if I am being irrational.

Answer 1

I think you have this all correct. The world would be a better place if all papers released all of the software that is used to generate the results shown in it. And that's not just a personal thought of mine (and apparently of yours as well), it is empirically verifiable: Papers that release the software used get more citations than papers that do not -- in other words, others are also thinking that that is worthwhile.

People have all sorts of reasons not to release their software, including (i) they believe that they have a competitive advantage by keeping their software to themselves, (ii) they do not trust their own software, (iii) they do not comment their own software or otherwise use good software engineering practices, and don't want the world to see so, (iv) they do not want to provide support to others who would download the software and use it. I believe that (iv) is a legitimate reason. (i) is misguided in my opinion, because at least if the software is non-trivial, others trying to use it more likely than not will ask to collaborate with the authors of the software, rather than just use it themselves; in other words, the original authors would gain a competitive advantage rather than disadvantage from releasing the software. (On this point, I speak from many years of experience.) Finally, if someone does not want to release their software for reasons (ii) or (iii), I believe this is an ethically questionable approach: If you have no confidence in your work, you probably shouldn't publish it.

In any case, I believe that your arguments are correct. Whether they convince your adviser is, of course, a different question.

Answer 2

Check out this famous paper (in AI) by Google. It described an algorithm by which an AI can learn to play a game - in this case chess and shogi - by playing against itself only, and how such an AI can beat the then-strongest engines at the games.

Google didn't release the source code. But! The papers contain enough information for someone else to duplicate their work. Using the same methodology, the chess engine community created Leela Chess Zero, and tuned it (Leela is likely stronger than AlphaZero at this point). The ripples of the new AI continue to be felt, since 1) it affected how Stockfish, the strongest traditional chess engine, was developed; Stockfish is likely stronger than Leela again, and 2) the gameplay ideas that Leela discovered (and continue to discover) have been incorporated into the repertoire the top human grandmasters.

Now we can ask the question: what good is this paper if the code is not open source? How do we know Google didn't fake their results? I'm sure you can see the answers in the second paragraph above.

Answer 3

While I won’t comment on your particular case, I think it’s useful to consider science as a whole. You should realize that computer scientists are uniquely able to distribute an entire exact experiment implementation broadly, instantly, to anywhere in the world by virtue of making the source code public. Anyone can reproduce such an experiment exactly, and this is (in principle) a benefit. However, for any physical or biological experiment, there is no way to transport the apparatus to everyone. The journal text, figures, published data, etc. are all anyone can hope to have to learn of an experiment and judge its conclusions and correctness. There is an implicit assumption (barring evidence otherwise) that the researchers competently performed the experiment and collected the data they describe. Trust is a fundamental component of scientific communication, and this is why malfeasance which abuses this trust (such as data fabrication) is taken so seriously.

All of this is to say that every other experimental science views journal articles, with all their flaws, as having value; they have been the primary vector for dissemination of cutting-edge results for well over a century. They work because there is generally a strong culture of scientific integrity among scientists, and people for the most part take the presented data at face value even though the experimenters could be incompetent or untrustworthy. So although you enumerate reasons why journal articles are to be mistrusted, they have served science through the period of the greatest expansion of knowledge in human history. And they shall serve you just as well.

Answer 4

Going point by point through your issues...

The papers do have some theoretical content, but they do not seem to be particularly valuable. Anyone else could have come up with those ideas with a little bit of thought.

This is undervaluing your contribution. Your paper, by definition, does have some useful theoretical basis otherwise you wouldn't be publishing it. It may seem obvious to you, but that's likely only because you've been living with this for a while. And I'll remind you that Thomas Henry Huxley's response on first reading Origin of Species was "How extremely stupid not to have thought of that!" What's apparently obvious after you know it may not be obvious before.

If there are no executable artifacts tied to my paper, I could just have been lying about my results. I am not, and the experiments are actually very rigorous, but this is fact is somewhat undermined

And this is why it's important for other places to independently check results. It's not sexy, but it catches dodgy results produced through incompetence or malice.

Considering that a large part of our experimental work depends on comparing with artifacts created by other research groups, it seems petty not to release our artifacts

And if they've got the same dataset as you and the same code, they'd get the same results if they just ran your code. The important thing isn't the code, it's the algorithm. Which is NOT the same thing.

The potential benefit of the transparency of having open source code seems to outweigh the potential harm from potential embarrassment that may result from bugs found by the community. In my experience, artifacts released by other research teams in similar areas do often contain bugs, but I still think positively of them.

But what if the "interesting" feature of your results is actually a coding bug? Someone simply blindly copying your code will get the same results. Independently coding the algorithm though is unlikely to get the same bug twice, so they can report that they don't get the same results and your paper may be incorrect. Having the source code would let them analyse why your results were dodgy, sure, but that's less important than discovering the fact of it being dodgy. And more likely, they'll be in contact with you (or your supervisor) after they discover they can't reproduce it, and then you'll be checking your own source to find where it went wrong and publishing a correction.

Answer 5

The goal of a paper is to report a novel research finding. It can accomplish this perfectly well without providing source code.

It is not possible to describe empirical findings in full detail in a scientific paper (or arguably any format). It ultimately falls on the author to decide what detail is appropriate.

The scientific method is not about believing observations because they have been recorded in excellent detail, but about independently verifying them. So it doesn't particularly matter whether you provide the implementation code. What matters is whether it's clear to other researchers how they would implement their own version such that it confirms the results you obtained. Your code may or may not be an essential aid for that.

Some, including me, would argue that the virtue of a scientific finding is simplicity. In your paper, the more quirky things there are that have to be done "just right" - whether the algorithm works when implemented in a certain way with a certain language, or it only works on a certain kind of input - these all make the finding itself less interesting to begin with. So if your paper is really good enough to care about, the code is a moot point. I should be able to write my own code from scratch and verify your algorithms and theorems.

However, beyond the primary goal of justifying a claim, there are secondary goals that are well-served by making the implementation available:

• If your implementation has a mistake, it will be found sooner
• If your implementation turns out to be correct but hard to reproduce, you will not be accused of fabricating results
• Researchers wishing to extend on your work will have a better starting point
• Non-researchers (such as those in the industry) wishing to apply your findings will have a better starting point
• People trying to learn about research in the field will have better resources
• The code will serve as a public demonstration of your coding ability, if required by for example job applications

I personally would prefer if code was always published along with papers, but there have been many papers in CS and other fields that were published without code, and yet their claims are sufficiently credible and the findings they report are useful, so I would not consider it a hard requirement.

But to your concern about the code being unpolished: It doesn't really matter. No code is perfect. Either what you have works which means it's good enough, or it doesn't work which means your paper is wrong in the first place. You have little to lose from publishing code.

You also mention publishing binary executables, which is the really strange part to me. By publishing executables, you are asking people to trust that you did not introduce any malicious code (like viruses) into the binary, and also that your system and every other system in the chain (like the CDN actually hosting your binary) is secure from hackers and so forth. This is a very big claim, and completely unnecessary when you could just provide source code and completely obviate it. You are also training users to accept bad security practices. People who do publish binary code (eg. proprietary vendors) will at least take steps to mitigate the security hole, by employing security engineers and signing their binary builds. If your goal is to let people use your implementation without seeing the code, you should provide a web server so that the binary is executed on your system, or at least containerize it and provide something like a Docker image.

Answer 6

Unpublished papers, in and of themselves, are almost entirely worthless, like an unfilled lottery ticket, or a movie script that's never been filmed.

Once published, they get a little value, like a lottery ticket that has had the numbers filled in and registered at the checkout, or a movie that has been shot. They can all now be tested, and have a chance to win.

The community ascribes them value at this point, but this is equivalent to the face value of the scratchcard (a dollar, perhaps): there is a chance that the paper might turn out to be a "winner", based on correct data and correctly applied algorithms. Every panel on a scratchcard could be a winning number... or none of them may be. A movie could be a blockbuster... or a flop.

Reproducing the results is the proving ground: it's how they provide value to the consumer, like when lottery numbers are drawn, or a movie is released to the public.

The higher you make the bar of reproduction, the lower the chances of making any valuable contribution to the community.

The lower the bar, the more likely someone reading the journal you publish in is to say "hey, this is something I can give to my student to try" or "that's so easy to reproduce, I could try that out this lunchtime..."

There are those who argue "if you release the source, then people will run it and necessarily must reproduce your errors". This is very twisted logic: releasing the source saves them your entire development-cycle of time, meaning more people can work on what your paper gave. More eyes on your source can only mean more likelihood of finding errors: it is impossible for it to mean less.

There are cases (like the Google AI paper mentioned in another answer) where papers that involved software have been reproduced without source code, and managed to be beneficial anyway... but these are far less common. Even in the Google case, there was only one project which went through the trouble of rewriting that code. If they hadn't got lucky, if that project hadn't picked it up and run with it... what value would that paper have had to the community, then? How much MORE value would it have had if instead, they had released the source, data, and a docker file, so countless people could have just run one command and reproduced their results? How much bigger would the field of AI be now? Would we all be driving self-flying cars? We can't tell, but we can guess that the impact of the paper would not have been at all reduced by improving the reproducibility.

Assuming someone DOES reproduce your work, but gets different results... where is their error, if any? Even if you collaborate, and both use the same data to get different results... is it their error, or yours? How do they find out? What if you are not contactable? In software engineering it is considered a truism that you cannot easily identify or fix an error without reproducing it first, and if your output cannot be reproduced, it cannot be fixed.

TL;DR: All papers have some value, but without reproducibility, that value is minimal and aspirational, rather than real. Maximizing reproducibility maximizes real value and impact. Source code really helps these goals.

Answer 7

In Natural Sciences we should put forward boldly our hypotheses and results inviting debate, correction and, ultimately, falsification. Do publish your code. Why not clean it up, while are you at it? If for no one else then for yourself, who is the one most likely to re-use the code in the future.

Answer 8

The potential benefit of the transparency of having open source code seems to outweigh the potential harm from potential embarrassment that may result from bugs found by the community. In my experience, artifacts released by other research teams in similar areas do often contain bugs, but I still think positively of them.

Let me answer from a software developer's point of view. Open sourcing the code has a huge benefits, even if it contains bugs or poorly written code. Point of being opensource is other people can contribute towards it. The code will be improved by others, bugs will be fixed and will be much more popular compared to a closed source one.

So, if you are confident on results, why hesitate?