25

I am a PhD candidate working with a large computer code (my code plus others, well over 100,000 lines). I am starting to realize that the code will always have bugs in it, no matter what I do. I'm in engineering, not computer science, but it seems this is just the norm for software development. Despite the bugs, I've been able to do real research with this code and demonstrate viable results.

Nonetheless, I have a mental list of all the bugs I haven't had time to fix, and I've been under the impression that I need to write about them in upcoming publications for the sake of being transparent. Yet when I draft some words to describe / explain them, it seems really silly and distracting from the main point of the paper... these bugs don't impact the main conclusion of the work, so why tell the reader something that doesn't even matter?

What do you all think about this? Do you describe every known bug in your publications? Or can some of these things be omitted for the sake of clarity?

Additional info: The research is to study a new method for design optimization of an aircraft. The bugs are flaws in the engineering test-case set-up. In other words, the code is not as accurate as it should be if this aircraft design were to be built -- but the results do without a doubt demonstrate the optimization method. i.e. the code was successful at optimizing the design, and the test-case is accurate enough that it represents a real engineering problem. There are bigger approximations regarding the aircraft design that are intentionally used to simplify the problem. These have more impact than the bugs, and are clearly discussed in the paper, so no one will ever take this as a real design and try to build the aircraft.

14
  • 13
    Do the bugs actually affect the results (even if they don’t affect the conclusion)? Apr 3 at 6:32
  • 10
    Even assuming it doesn't affect your results/conclusion (I would like to hear more about how you know this for certain), it seems a little strange trying to justify buggy code in a paper instead of just fixing it. If you need help, you could request your advisor for support -- there are probably hundreds of very capable undergraduates in your university who would jump at the opportunity. Apr 3 at 7:16
  • 11
    Also, I just have to say this as a (former) software developer: while bugs are exist in any software, they are not the norm. Teams of developers work very hard to fix them; and although even good software has bugs, those are often complex edge-cases that are difficult to isolate. This is very different from your situation where tech debt seems to be passed down to the next graduate student with no responsibility for fixing them. Apr 3 at 7:20
  • 3
    Are you and others maintaining the code? If so, you should have a mechanism to report, track and fix bugs, which I would say does not need to be in the paper unless it is affecting results in the literature/current research. You also probably want a mechanism to independently verify your results in some cases, as cross-checks on your results. This is good to mention in the paper.
    – Kimball
    Apr 3 at 12:43
  • 4
    Is the bug a limitation of the code or a limitation of the method? In other words, is the bug caused by your research's intellectual contribution having a flaw/limitation? Or is the bug caused by bad code style, version/environment conflicts etc.?
    – Taw
    Apr 3 at 12:46

9 Answers 9

40

The most successful paper I ever published was a software tool. We released 9 years ago, preprinted 8 years ago and published 7 years ago. It has over 1000 citations and 100,000s of downloads.

Since publication we have closed over 400 issues in the GitHub issue tracker. These won't all have been bugs - many people misunderstand how to run the software, but there have been 900 commits. We currently have 11 issues open.

I don't remember how many issues there were open when we published, but I doubt it was zero. And given that issues come in at about 2 a week, if it was empty when we submitted the publication, it won't have been empty when it was accepted.

We wouldn't have published if we had known that there were major bugs that made the software give seriously erroneous results, but we knew there were things that were sub-optimal, data types it wouldn't work well on, install dependencies that were fragile, etc. Given that we've had about 2 issues a week since we released the software until now, and that hasn't really sped up or slowed down, there would never be a time when we could release it 100% bug-free. We published when we were confident that we had something that produced results that were provably better than what came before and could be validated by orthogonal means.

As the paper was more about the abstract algorithm than the implementation, we didn't talk much about the known implementation problems in the paper, although we did talk about the limits of data where it performed well and where it didn't. Limits that we've improved since publication.

20

I am starting to realize that the code will always have bugs in it, no matter what I do...it seems this is just the norm for software development.

This is troubling. It's true that few things are ever perfect; it's usually possible for code to be improved. Perhaps it could be faster, or more robust, or make fewer assumptions, or be more accurate. But buggy normally refers to actual errors: either the code doesn't run at all, or it doesn't reliably do what you think it is doing. If buggy is truly the correct word for what you mean, I would not say that this is normal.

my code plus others, well over 100,000 lines

I think this is the real issue. It sounds like generations of students have been using a semi-shared codebase with poor practices, and now there is a mountain of technical debt. I empathize that you don't want to be the one to spend months fixing it all up. But I cannot in good conscience advise you to publish with buggy code. Rather, I hope your advisor will recognize the problem, distribute the work, and set up better practices going forward.

Nonetheless, I have a mental list of all the bugs I haven't had time to fix

Sounds like you need an issue tracker.

these bugs don't impact the main conclusion of the work

How do you know this is true? This is the fundamental problem with bugs.

  • If you fully understand the problem and know how to fix it, then maybe you're right that it doesn't impact the conclusions. But in this case, why not just fix the bug?
  • If you don't fully understand the problem or solution, then you can't possibly understand all the consequences, and so it's possible that it does impact the conclusions.

I've been under the impression that I need to write about them in upcoming publications for the sake of being transparent.

It depends.

  • You do not need to go through all the gory details of your implementation. Ideally, you would make the code publicly available, and the issue tracker can list any known issues.
  • If these bugs could impact your results, then you do need to disclose them as you say. But in this case, I would expect reviewers to tell you to fix your bugs and then resubmit.
  • The exception might be if one failure mode is legitimately hard to fix. In this case, you can say something extremely brief like "for transparency, we note that our basket algorithm has a known bug with its underwater weaving algorithm. This seems to be related to an issue with one of our dependencies [X]. However, our analysis shows that this affects the presented cost-depth curves by less than 1%." But this is unusual and should be reserved for really intractable bugs, not just bugs that you don't feel like fixing.
9
  • 16
    "If buggy is truly the correct word for what you mean, I would not say that this is normal" I have yet to see a single non-trivial software that is free of bugs. That includes even security relevant things where millions are spent on validation like flight software. Claiming that that's not normal seems very weird to me.
    – Voo
    Apr 4 at 10:22
  • 12
    Masters in Computer Science here. If its bigger than Hello World, it has bugs. Some Hello Worlds have bugs too.
    – T.E.D.
    Apr 4 at 13:51
  • 5
    @T.E.D.: Case in point, in your comment you are missing the comma in between Hello and World! (This is not meant to be snarky, rather, it is meant to show how hard it is to have software, or really, anything, that conforms 100% to its specification … and there's always the possibility of bugs in the specification.) Apr 4 at 15:28
  • 4
    @JörgWMittag - "and there's always the possibility of bugs in the specification.". <-- Exactly. Hence the famous Knuth quote: "Beware of bugs in the above code; I have only proved it correct, not tried it."
    – T.E.D.
    Apr 4 at 18:35
  • 5
    "reliably do what you think it is doing". A lot of bugs take the form of, "this usually does what I expect it to do, but there's an edge case where it doesn't." Such bugs are very common. Whether that counts as reliable or not I guess is subjective
    – T Hummus
    Apr 4 at 20:56
17

Usually, research software's existence is aiming to solve some scientific question (I'm assuming you are not writing a paper just about a program, which some journals do accept).

Given that, there are two cases:

  1. The bugs don't affect the scientific results.  In that case, there is no need to discuss them in a paper, because people who read your paper don't need to know about bugs that don't affect the scientific conclusions.
  2. The bugs do (or you suspect might) affect the scientific results.  Then they need fixing, the same way you shouldn't publish the results of an experiment you seriously suspect are wrong because you ran your experiment wrong.
5
  • 2
    "The bugs don't affect the scientific results" -- But if you assume that, you'd better be able to back it up. If you genuinely understand a bug well enough to know all the effects, you probably also understand it well enough to fix it.
    – Ray
    Apr 3 at 16:55
  • 2
    @Ray Not necessarily. The bugs might be in how command line arguments are parsed. Or they might be in the build routines. Or in setting that allow the software to run on a datatype that isn't used in the published results. Apr 4 at 8:02
  • 4
    I have lots of bugs I understand but can't be bothered to fix in research software I work on, some examples: * Terminal output is badly messed up when software is used in parallel mode -- workaround: Use the "write final answer to JSON" option. * When using the 'kissat' option, I never free memory as search progresses, so for larger problems we will exaust the memory and the software crashes. workaround: kissat doesn't work for large problems, but it's super fast so we provide the option. I'd call both of these bugs, and I hope to fix them one day. Apr 5 at 1:40
  • @ChrisJefferson: "I never free memory as search progresses" <-- totally unrelated to OP. Why? What could've motivated you to neglect memory management? Like is it intricate in some way? Genuinely curious.
    – Argyll
    Apr 5 at 13:36
  • I'm using an external program which I converted to a library. The program didn't clean up after itself (because there is no need to when a program finishes), so my conversion, currently, doesn't either. I can "restart" the library, by calling it's init functions, but these just abandon all the old memory. Fixing this will involve adding tracking to collect all the memory I'm losing. Apr 6 at 10:21
7

Most programmers would agree that all nontrivial software has bugs. Writing bug-free programs is the kind of absolute perfection humans aren't capable of. So it's not necessary or helpful to say merely "the software has bugs"; this can be taken for granted. Nor is it probably helpful to list every bug in your paper, because it's unlikely that every bug is germane to the results you're presenting. If one or more bugs are germane to your results, then by all means you should mention them, perhaps in the context of other limitations of your study. If you want to be especially thorough and transparent about bugs without burdening your paper with a full bug list, use a web-based issue tracker (or just write up the list of known bugs as a web page) and link to it from your paper, with a few words about the most important bugs.

3
  • 2
    The only developer I know of with a reputation for "no bugs" is Donald Knuth himself. And even TeX had bugs. But by version 3, he decided it was in pretty good shape. So the next bug fix made version 3.1. The second bug fix made it 3.14. TeX is now on 3.141592653. The fact that we can call this piece out as the exception really does prove the rule.
    – Cort Ammon
    Apr 4 at 4:58
  • 2
    @CortAmmon And TeX is a pretty small program - TeX82 in its third stage had about 14k pascal statements. And even that little program, written by one of the cleverest humans around had 440 bugs published. And that was a rewrite of an existing system where lots and lots of bugs had already been found. And despite all of that it still had a bug in every three statements. The amount of errors in most modern software will be orders of magnitudes higher.
    – Voo
    Apr 4 at 17:51
  • 1
    @Voo+ 440 / 14k ~1 / 31.8 -- slightly less than one bug per screenful. Now, what's the bug rate for SO comments <g>? Apr 5 at 2:31
4

Are other people going to use the code after you?

If it's purely a personal thing for your degree, then don't worry about it. Chances are, no-one (including you) will ever look at it again. This might be a harsh truth now, but in 5 years time when you're in industry doing some hundred-million-dollar project with 200 other people, that project will be far more interesting.

If it's something which people are going to be carrying on developing after you leave though, none of this should be merely "in your head".

If other people will be working on it, or are already working on it, you need an issue tracker. Stat.

Maybe those bugs don't affect you right now - but they might be fatal for the person following you. The more info you can give them, the better that handover will be.

They might not even be your bugs - I note you said that multiple people had worked on it - so you're not setting yourself up to be judged here. And if you've got multiple people still working on it, an issue tracker can be a good way to see who's working on what, as a way to manage division of labour. An issue tracker can also capture the extra features you haven't had time to do yourself, but someone else might find useful in future.

The final reason to get into the habit of using an issue tracker is that every competent company in industry, and every open-source project, uses issue trackers. It's simply best practise, and the sooner you get used to filtering everything through formally-tracked issues, the easier your future career is going to be.

2
  • Indeed, I think the most worrying aspect of the scenario is that the list of bugs is a "mental list". Ideally both the code and the issue list should be in GitHub, and the thesis should include a reference to the commit state of the repository from which results were produced, so anyone can download that version, reproduce the results, and check for themselves whether any of the known issues affect the outcome. Apr 4 at 23:22
  • @MichaelKay Exactly. I don't know whether this gets taught at uni these days in software 101, but I could well believe that this is something they could easily miss.
    – Graham
    Apr 5 at 7:19
4

For research code, one good practice is to not only prepare many tests to check individual submodules, but to run integrative sanity checks on known cases.

These can be either well-understood models, parity or other checksum tests (e.g. energy-conservation if you are in physics, accounting balance if you are in accounting, function/loop invariant if you are in computer science, etc.)

Where this is not possible, you need to have an idea of the potential errors, a standard practice in numerical analysis. Some well-used algorithms did, for a long time, not have good error estimates (e.g. 4-th order Runge Kutta), but were used extensively due to good experience values.

In any case, you have to put substantial effort to have a clear error model about when your stuff can fail and how to check this.

It is your responsibility that the results you publish represent your best-faith belief that they are correct.

One example is that, if you code may occasionally crash, but when it runs, it always provides correct results, that may be buggy code in the sense that it cannot deal with pathological cases, and is not suitable as user-friendly code for 3rd parties' use, but is acceptable as research code.

But you must be able to trust that the results represent what you claim they do. This is on you. Of course, errors are always possible, but you need to exert the best-faith effort to make sure that they have no role in the results you see.

A rule of thumb that I particularly like is: in the special case that your results are just so perfect, be as distrustful about them as a conspiracy theorist is about the government and the big corporations. Run sanity checks back and forth to make sure things really fit together at all levels.

The nastiest-to-debug cases are randomized algorithms, e.g. Evolutionary Algorithms. Here, systematically debugging with seeds, can help you find problems.

Other nasties are algorithms that depend on race conditions in parallel threads. Another vexating example is one where I once spent a whole week trying to find why my dynamics started out the same, but then diverged after 10 or so integration steps, depending on the optimization level of the compiler. Turned out my system was chaotic, so the dynamics indeed was sensitive to the minuscule numerical rearrangements that were induced by the higher optimization levels.

3
  • 1
    Your second paragraph makes me think you are talking about some subfield here ... Apr 4 at 2:51
  • Captain, pretend you are a competent scientist who doesn't know what "parity or other checksum tests (e.g. energy-conservation if you are in physics, accounting balance if you are in accounting, function/loop invariant if you are in computer science, etc.)" or "4-th order Runge Kutta" or "Evolutionary Algorithms" or "race conditions in parallel threads" or "dynamics started out the same, but then diverged after 10 or so integration steps, depending on the optimization level of the compiler" or "minuscule numerical rearrangements that were induced by the higher optimization levels." is and Apr 6 at 15:03
  • ... re-read your answer and ask yourself whether it is understandable. Apr 6 at 15:04
3

Ahh, the joys of a legacy code base!

Others have answered about the publication aspect. I will address how you save your lab mates from the same problem in the future by writing tests.

IMHO, an Issue Tracker is necessary (as mentioned by @Graham and @cag51 above), but not sufficient. It doesn't tell you before you release if you've re-introduced an old bug or created a new one.

Some people say that Test-driven Development takes the surprise out of programming, but a few tests will start to give you the confidence to fix those nagging bugs. Don't go overboard, but do a little at a time when it will benefit you. I started using it in a highly disruptive environment just so that I could run a small portion of the code I was working on in isolation.

Perl uses the Test Anything Protocol and I love its todo function that says "Don't complain when this test is broken, but tell me when it starts working". Other languages will use xUnit or something like it. Nothing beats starting the test suite in the morning, making a coffee and sitting down to a green line saying All tests successful.

Please, commit to fixing ONE bug. Document the proceedure of writing a simple test and how to run the test suite. Make it easy so there's no excuse not to run it. This one act will save you 6 months later and you will understand the your test suite's power for good. Then you will start telling anyone who will listen how to run your test suite and write their own tests. Eventually the bugs in your code base will diminish as you only have to fix something once and the next generation will sing of their deliverance from development hell by The Magnificent Katie9206.

Living with known bugs should not be the norm for software development. The Society for Research Software Engineering promotes software skills in research and may be able to put you in touch with people who can advise.

2
  • It seems to me highly unlikely they have built a system with 100K lines of code without some kind of test framework. But who knows, you might be right. Apr 4 at 23:26
  • I've seen CompSci academics I respect release 7k loc without a single test. I expect most people test by hand what they've added/changed, but never consider the action-at-a-distance bugs that go unnoticed. Large organizations have learned to automate tests. Individual labs write code out of necessity and can remain oblivious to lessons learned elsewhere. I really hope I am wrong. Any horror stories out there?
    – Boyd
    Apr 6 at 16:16
2

What are you trying to accomplish?

None of the other answers are relevant to engineering: First and foremost, you must have some goal or set of goals (requirements) you are trying to achieve. What you describe as bugs might be anything from non-starters to non-issues, depending what your goals are.

If, for example, your requirement is to optimize that specific design, software bugs are totally irrelevant: if you have a demonstrably optimized design in hand, you're done. It doesn't matter whether the tool is good because the tool is a means, not an end.

If your requirement is to demonstrate the method, bugs are only relevant to the extent that they affect that demonstration. For example, if you have horrible code that randomly formats your drive, that's pretty unusable as a practical tool, but might be totally fine as a demonstration of a method. Less ridiculously, you might have a bug where not following certain input conventions causes the tool to choke, and you manually sanitize your inputs: it's a non-issue, not relevant to your requirement.

If your requirement is to create an actual commercial-equivalent tool, then security holes you would not have even otherwise considered are likely show-stoppers.

Etc.

So, figure out what you are trying to accomplish first, then you can figure out which items on your mental list affect your result.

0

TL;DR

There's no need to burden the main text of your paper with details that don't affect the results, but the known bugs should be documented (in appendices, code sharing platform, and / or the code itself).

Share the code

First of all, you are sharing the code publicly, right? If not, you should. Evaluating or building upon a piece of research that relies on 100k+ lines of code without having any access to that code is basically impossible.

Don't keep the bugs to yourself

The bugs that you know of should not live only in your head. They should be documented in at least one (preferably more) of three places:

  1. In the codebase itself. Hopefully each of the reusable pieces of your code (functions, classes, etc.) have attached docstrings. You should point out any known limitations there. For instance, Using float64 here is unreliable and should be avoided. It's best to be conservative here, so maybe even better is, e.g., This is only tested with float32.

  2. In bug/issue trackers. Given that you're sharing the code, you are presumably using a platform such as GitHub. These platforms usually have embedded issue or bug trackers: take advantage of them! Post any known issues there.

  3. In the paper's appendix / supplementary material. I think your intuition is right that you don't want to spend too much space in the paper describing bugs that are ultimately tangential to the point of the paper. However, most venues allow for appendices / supplementary materials, and those would be great places to describe the known bugs and / or link to the issue tracker (see point 2 above).

It's OK not to fix everything

Any project needs to end at some point and if the project is big enough, there will be known issues that you won't have time to fix. That's OK, as long as they don't affect your results. Do make sure to document the known issues, though.

In the paper you can include a sentence along the lines of

The focus of this paper is on the optimization method, and we use a toy model to demonstrate its effectiveness. Note that this model has a number of shortcomings, as outlined in section XXX, which make it unsuitable for real-world use. We believe however that it is a good test case for the optimization procedure.

Then in "section XXX" you describe both the approximations you mentioned in your question, and link to the appendix section describing the other known issues.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .