Best-practice models for "research" code?

Question

I have been a professional software developer for a number of years, I'm also an academic researcher - and my research has involved lots of software development.

I sometimes feel as though my industrial experience has been a hindrance in my research, as the goals of writing software in a research context feel contradictory to the goals in industry.

In industry, code needs to be (ideally): maintainable, bug-free, refactored, well-documented, rigorously tested - good quality - best practice says that these things are worth the time (I agree).

In academia, the goal is to write as many quality research papers in the shortest possible time. In this context, code is written to run the experiment, and might never be looked at again (we are judged on our papers - not our code). There seems to be no motivation to write tested, maintainable, documented code - I just need to run it and get the result in my paper or whatever ASAP. Consequently, the "academic" code I've written is poor quality - from a software engineering perspective.

The problem is that I either spend too long making (unnecessarily) getting my "research" code to industry-quality, or I publish work based on "bad quality" code, and I feel like a fraud.

My career progression is dependent on me writing "bad" code!?

The "craft" of software development is a huge subject - but where is the best practice for academic research? Nobody writes unit tests for conference paper code!

Does anyone find them in a similar situation? Does anyone know of formal methodologies for "research" code?

Answer 1

I think the key to understanding research code for industrial software engineers is to accept that you are typically not building a product. You do not have customers as such. You are building software to prove a point.

As such, the majority of code that you write as a researcher is more akin to the throw-away prototypes and mockups that you (in industry) often write in the early phases of a project. As you are certainly aware, even in industry these mockups have quite different properties than the final software. They primarily need to:

• Have exactly the features that you want to show to the customer. Not more, not less. Typically, all the boring standard features for the domain are omitted.
• Need to get done quickly. Both you and the customer know that the prototype will be thrown away anyway, so it does not matter whether the code is maintainable.
• Need to be easy to extend and adapt, optimally live during the demo.

Essentially the same properties are also useful for most throwaway research code. You do not want to build features that you do not need. You do not want to waste time writing e.g., maintainable code if you know that it will not be maintained. You want to use an environment that reduces the amount of boilerplate code and setup, and which maybe auto-generates a lot of code for you that is "good enough" for your demonstrator (Ruby on Rails and its scaffolding features come to mind).

My career progression is dependent on me writing "bad" code!?

No, it depends on you writing code fit for purpose. Just like in industry. In industry and academia nobody applauds you for software qualities that are not needed. Try to reconsider what the point of the code is that you are writing. If you plan to release your code as open source software and you expect it to be picked up by other people across the world, then go nuts - use all the engineering techniques you have also used in industry to build the best product you can. If your goal is to evaluate this one algorithm or principle for your conference, and then throw away the code, then you can also live happily without writing a single unit test without feeling like a fraud at all.

EDIT: of course this does not mean that it is acceptable to write code where you are unsure whether you have implemented said algorithm correctly. Ensuring that what you have indeed shown what you claim to have shown is mandatory, especially in research code.

Answer 2

I am a researcher and self-taught developer. I have done substantial projects which were primarily software based. Although my work is far from the most "hardcore" stuff that's out there in terms of complexity and scale, the projects were big enough that naive mistakes (eg. not using version control or poorly documenting code) were very painful. I ended up learning quite a few "best practices" through trial and error.

I have also been on the receiving end of "unmaintanable code passed down to fellow researcher":

my industrial experience has been a hindrance in my research

No, you have basically come from a civilized environment that solved these problems decades ago into one that is stuck in the stone age in terms of software development hygiene. Scientists still code like it's the 60s. Of course you feel a conflict, but the fault is not with you.

In industry, code needs to be (ideally): maintainable, bug-free, refactored, well-documented, rigorously tested

Let's say the speaker at a scientific conference, while describing the computational part of his research, said one of the following:

"The code I wrote for this research is, admittedly..."

• ...unmaintainable (and good luck building on my research!)
• ...full of bugs (and I have no idea if the output is even correct!)
• ...unreadable spagetti (and I don't even know how it works, let alone if it does so correctly!)
• ...undocumented (and all the mistakes are obfuscated from reviewers!)
• ...not tested (so god knows if it does what I say/think it does!)

Do you expect the audience to react with anything but scorn and outrage? If I heard such a thing, I would not believe anything this person published ever again.

In academia, the goal is (...) in the shortest possible time.

Yes, but "no shorter". You don't skip vital control experiments because "controls take time". You can't skimp on code quality for very similar reasons.

There seems to be no motivation to write [good code]

Because this is an endemic problem of academia. Although computers have been used in science for decades, it seems that algorithms have only become an important part of research in the last decade or so (perhaps because of "big data"). When you base your research on code, that code must be good quality. It is not enough to simply crank out some buggy write-only script and call it a day. The software development community has figured all of this out long ago, but academia has not yet caught on - I think the reason is that most scientists do not have a formal background in software development, and there have not been enough huge scandals in research caused by bad programming practice (eg. key results of a high-profile paper turn out to be artifacts caused by bugs).

Consider how, in many disciplines, reviewers will not even ask about the source code of your computation-heavy paper. How can they evaluate, then, the validity of your results? They cannot, and this is a failure of the peer review model as it currently exists.

Sorry to go on a rant, but basically, it's like this: As you know, there are very good reasons for writing quality code, even if no one is watching over your shoulder. In science, currently it so happens that nobody cares if your code is good or not. But this should not be a reason for you to not write good code anyway - the reasons for writing good code in the industry still largely apply to science.

Unfortunately, you may not be rewarded for your extra work. You may even be punished, because as you say, good code takes longer, and others may not see beyond that. Your PI or colleagues may not understand why you are so much slower. The best you can do is explain to them the need for good practices.

Obviously, there are exceptions. For instance, you may not need to worry about portability or backwards compatibility with old versions of the OS for code that is meant to run on a dedicated lab computer (although it is undesirable to write your code such that it only runs in a very exotic environment that other scientists will not be able to easily reconstruct). But by and large, I find that industry practices still apply, and the exceptions can be easily detected by applying a modicum of critical thought. That said, there is also a helpful publication called "Best Practices for Scientific Computing" which examines this matter in detail.

Ultimately, it is an ethical decision you must make. Do you care about doing good science above all else? Follow best practices. Do you want to cut corners that you shouldn't (in an ethical sense), to save time or avoid friction with co-workers? I couldn't recommend you to do this, on principle. But obviously many people do, and perhaps in practice, some scientists are forced to do it - although then again, does being unable to do good science by circumstance excuse bad science?

Also, like I said, I think part of the problem is that there haven't been any big scandals. If you do skimp on code quality, there's a chance it will catch up with you. You might even end up being one those big scandals. Admittedly, the risk is probably small... But, I think you can see my point.

Answer 3

I either spend too long making (unnecessarily) getting my "research" code to industry-quality

Don't. Make it as good as possible within the suggested timeframe. Aiming for 100% perfection that will require double the time is not worth it. In this sense, research is exactly like industry.

Consequently, the "academic" code I've written is poor quality

That does not mean all academic code is of low quality. If you check papers on algorithms conferences or parallel processing systems, you will see that the developers have thought even excruciating details, like reordering of data for fewer cache misses, SIMD, GPU programming, SSD storage etc. Usually advanced CS algorithms research is some years ahead in adopting new methods, hardware techniques before any of those techniques actually hit the industry. On the other hand, in more theoretical CS conferences code is mainly a tool and as such, it does not have to be cutting edge. So, the quality is related to the audience of your product code (exactly like industry).

Does anyone know of formal methodologies for "research" code?

I have never heard on any methods especially tailored for research code. Still, you can use the practices from your industry background (when they actually accelerate your process of writing the software). For example a versioning system accelerates development and minimizes errors / losses of data. On the other hand UNIT tests take a lot of time, which you might not have. A informal wiki for bugs, documentation, features might worth the extra time, since it also accelerates writing the actual research paper. Contrarily, a full blown bug database (bugzilla) might not worth the extra time and effort.

So, stick to those industry methods, techniques you know will save you time on the long run and will improve your software but without taking all of your time. Finding a middle ground is always the best solution.

Answer 4

tl;dr Some parts of industry "best practices" fit in well, and other parts are inefficient in a research environment. Keep what works well in this environment.

---

I'm a experimental particle physicist and we write a lot of code and much of it is big projects written by many, dispersed programmers.

Some part of the usual industry tool kit we use enthusiastically

• Version control
• Bug tracking
• Automated build and test systems

Other part are either slower to catch on or not as highly prized, including

• A formal Process (with a capital 'P') with regular planning meetings and release checklist and so on. These appear as projects get bigger (usually in response to a total break-down in quality control or long release lags). That is, we use them when we need them.

• Documentation generation systems are pretty common but only lightly used until the project gets big when people who are forced to decode some bits often contribute a little more documentation.

• Unit testing is sparse and usually concentrated on the lower layers of the systems, but regression testing is more common.

• Most projects have coding standards, but they are generally loosely specified and weakly enforced.

Other things simple don't show up much.

• Feature planning is pretty hard when you don't know what clever ideas a grad student will have next week to solve a problem that you haven't even noticed yet.

• It used to be true that strict check-in control got in the way of spreading experimental code segments around, and we'd simply freeze new development check-ins occasionally to get out a blessed release (a situation that rendered HEAD/trunk/whatever a "use at your own risk" proposition). With the rise of distributed version control there is starting to be a stronger commitment to check-in controls for the official trunk.

• Refactoring usually only happens when both a new person starts working with some old code and they feel they need changes or extensions. That which is not broken is left well enough alone.

---

From your question I suspect that you are doing your coding either single-handed or in a small group. In that setting the details change, but the tone remains the same. Simply keep the parts of your industry practice that work, and dispense with or delay the parts that have the worst cost/benefit ratio in terms of time/results.

Leave off the heavy refactoring until you know from evidence that a particular part of your code will be reused. Similarly, be content with rough documentation unless and until the code is shown to have a on-going life. And so on.

Answer 5

The main characteristic of research code (more so than typical programs) is that it is harder to plan. Research by definition reaches into the unknown, and this also translates in program structures. As a researcher you often don't have the time to refactor when your research slithers into a different direction that should translate into a different software structure.

In typical software engineering you (should) have a fairly good idea of the final functionality before writing a single line of code. In research, this is often not the case. Programming for research purposes is mainly about rapid prototyping, which is typically done differently than programming for long term use (e.g. little to no unit tests, use of different languages, ...). The main mantra is to get results fast, not optimal from a software engineering perspective.

Finally, proper software engineering is a fine art that is very hard to master. Even in industry settings, ugly software is abundant (when written by professionals!). The average researcher has no formal training in software writing.

My career progression is dependent on me writing "bad" code!?

As a researcher, you are not paid to develop software (unfortunately). As an idealist I like to believe that this will change over time, but for now funding sources only care about papers.

The "craft" of software development is a huge subject - but where is the best practice for academic research? Nobody writes unit tests for conference paper code!

Unit tests serve two main purposes: (i) assert that code is working correctly and (ii) find and debug problems fast, particularly after structural changes and refactoring (long term benefit for large infrastructures). As research software is typically fairly small, the first advantage is the only one that is really relevant. It seems that this advantage is either too small or is being underestimated (again, recall that most researchers are not software engineers).

Does anyone find them in a similar situation? Does anyone know of formal methodologies for "research" code?

If you want to change the world, start with yourself. I personally make a point of providing software along with a manuscript whenever it is reasonable. I also consistently ask for code as a reviewer, though this seems uncommon.

Answer 6

Rather than talking about differences with industry software development (which I have no experience with), I'd like to talk about things you should do in academia. I will be assuming that your code is not there for its own sake but attaches to some scientific statement such as "E. coli shares the foo-genome with humans" or "Algorithm A outperforms algorithm B in scenario Z".

• Best Effort

  While your goal is not productive use with revenue attached to it, you should have a reasonable amount of certainty that your code does what you claim and can be understood (peer reviewed!) by interested parties. That is, write clear code, comment and document. And (unit-)test.

  If nothing else, remember that you may have to revisit your own code some time later. You build a protoype but your next one may reuse parts of it. Or you want students to extend upon it.

• Accessibility

  In order to support scientific evalutation of your work (falsifiability, reproducability) other researchers have to have to be able to compile and execute your programs.

  Therefore, you should provide sources, build files/instructions and whatever input data you used in your work. Keep in mind that someone may want to build your program years after initial publishing, so make sure that the sources are still around then and the build process/instructions are reasonably robust against time (mention library versions). UI is not too important but "best effort" applies here, too.

  Consider putting your code on Github or a similar platform (e.g. your own). That way, you can publish updates and collect bugs easily. See also here and here.

• Licensing

  You should say something about what others (in particular fellow researchers) may or may not do with your code. You can use any license (I'd argue it should allow at least the liberties of GPL). The CRAPL might be worth a look.

• Fair Evaluation

  If your algorithm/code is the artifact you propose (as superior) you have to compare it to existing solutions. Make sure to use comparable input, rerun the experiment for the alternatives and follow basic experimental best practice (check out McGeoch's work, for instance). Make sure your comparison includes the accepted standard(s) of your field if there are any; if your approach yields different results, you have to explain why (that's okay/correct/better).

Accessibility is probably the most crucial item. Research code needs to be shared. I think this can't be stressed enough, and it's one of the major problems in all of computational science. Other than, say, physics, we have to opportunity to easily share and reproduce (most) experimental setup -- we have to make use of that.

Personally, I don't see why any article that founds its claims on computations that reviewers (and other parties) can not reproduce has any right to be published.

---

One reminder for all theory types:

Beware of bugs in the above code; I have only proved it correct, not tried it.
_{Donald E. Knuth}

Answer 7

I think your problem is in seeing quality as binary.

Whether in industry or academia we need to make choices as to particular quality goals. An ‘industrial’ piece of software might be safety critical and demand the highest achievable level of quality, or it might be an internally used development tool.

Quality goals might be:

• Uptime
  
  • Probably not a concern for you
• Robustness
  • Will you need to run with a lot of different data sets, or is it a one off?
• Re-usability
  • Will the processing be incorporated into the next problem you work on?
• Ability to verify
  • Does it work the way is it is supposed to? What would it mean if you published and this was not true?
• Ability to validate
  • Does it produce the right answer? [ Hopefully, this does matter. ]
  • Another way of checking the correctness of the answer might mean it does not matter if the logic getting there is wrong.
• Needs to be maintainable by
  • Self
  • Other expert
  • Some random undergraduate
  • Or definitely is throw away.
• Suitable to be made public for use by others to check or re-use your work
• Etc.

Having articulated your quality goals, your development and build system should be sufficient to meet those goals and no more. Note that this might vary from project to project.

Answer 8

If I could add a short version of the more accurate answers above, I would say:

• Academia -> prototypes.
• Industry -> products.

Products and prototypes must not be considered with the same standards.

Prototypes can have shorter documentation and may require some effort to be deployed. They can also be usable on a single operative system with very specific requirements, such as the ones where the code was initially developed.

Nonetheless, good quality prototypes should be tested, version-controlled, have a stable branch, and must have no out of date documentation. Otherwise they are just bad quality prototypes.

---- Edit

My career progression is dependent on me writing "bad" code!?

Leaving academia reminded me that exists something else than the paper-oriented working method, to which PhD students are trained (and whose outcomes have often a number of readers smaller than the number of co-authors).

It may not be the case in every academic environment, although in my experience research with scientific criteria is made in the private sector.

Also, this is not only a problem related to coding!

Some researchers working in wet labs have told me they have the exact same issue. Performing wet lab experiments documenting the full procedure, calibrating the tools to the highest standard, assessing the chemical composition of the material bought by third parties is consistent over time, making every step completely reproducible by another researcher... these basic practices resulted in a too slow academic paper production.

Answer 9

I was directed to this question by a chap in answer to my request for clarification "Why they were using a float type as a key to hash". Now to me this did not seem like a good idea. After a bit of back and forth, the questioner directed me here as their reason for doing it.

That was interesting as is the question itself.

Are you persuaded to write bad code in order to fulfill certain academic goals? Yes, even in CS courses I might add. At least bad in terms of comprehensibility. However as we are all too horribly aware, we have been 'persuaded' to write equally bad code in the commercial environment, or our lords and masters have been persuaded to have poor code written for them....

Leaving aside trivial implementations of algorithms. For instance should you use i and j as loop variables in a bubble sort algorithm, or indeed should you use a bubble sort at all. My response to your question, would be another one. How do do good software engineering principles help you achieve your goals?

Could, say good naming, SOLID principles, coherence and coupling et al get you to your goal more efficiently? My answer would be almost certainly. They are designed to do that in any non-trivial piece of software. It's what they are for, they were created in order to achieve software that could be changed at less cost. It doesn't matter that you aren't (or at least cannot foresee) a version two, the implementation isn't springing full blown from your mind, so it will be changing.

If you aren't sure what code you need to write, then something like TDD would also help given you have a ready made unit testing environment. Even without that "luxury" writing testable code is going to.

You have a huge advantage over your fellows who don't have a software engineering background, you should be able to get that irritating binary bit of the exercise out of the way quicker, get to the meat of your goal with less effort and then be able to expend more effort on the real goal.

I once had a discussion with an academically qualified type who told me re-factoring was just software aesthetes messing about and that I should have written the software correctly in the first place. Needless to say my respect for this individual dropped a notch or two, which was unfortunate for him as he hadn't earned that much in the first place.

So in summation I would say the sensible option would be to use good software engineering practices in all your efforts. Just to be clear though it is not good software engineering practice to write good code that you do not need...

To paraphrase Gandalf, "Keep it simple, keep it safe"

As for should you use a float as a key to hash to save time. If you know without any doubt as you choose that design compromise that the problems with doing so will not cause you an issue, then perhaps. But the amount of effort required to evade that compromise is fairly trivial and you just spent time asking how to get round an issue with using a float as a key. I rest my case

In any environment commercial or academic, choosing to lower quality, is both a benefit and a cost, examine both....

Answer 10

I have a problem with the premise of the question:

In academia, the goal is to write as many quality research papers in the shortest possible time.

There seems to be no motivation to write tested, maintainable, documented code - I just need to run it and get the result in my paper or whatever ASAP

I'm currently dealing with a large codebase written by people who apparently thought so, too. Before that I got used to a different standard of academic coding - one where code is simple, elegant, modern, well-tested and well-maintained over many years.

And I believe the people that will have an impact in this information age are not necessarily the ones with the most scientific credit, but the ones who can actually make something from their nice scientific ideas.

If however in your domain the cool things you can make don't depend at all on the code you write (e.g. just some boring data filtering), then you really don't need to bother with its quality.