Take the 2-minute tour ×
Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It's 100% free, no registration required.

I have been a professional software developer for a number of years, I'm also an academic researcher - and my research has involved lots of software development.

I sometimes feel as though my industrial experience has been a hindrance in my research, as the goals of writing software in a research context feel contradictory to the goals in industry.

In industry, code needs to be (ideally): maintainable, bug-free, refactored, well-documented, rigorously tested - good quality - best practice says that these things are worth the time (I agree).

In academia, the goal is to write as many quality research papers in the shortest possible time. In this context, code is written to run the experiment, and might never be looked at again (we are judged on our papers - not our code). There seems to be no motivation to write tested, maintainable, documented code - I just need to run it and get the result in my paper or whatever ASAP. Consequently, the "academic" code I've written is poor quality - from a software engineering perspective.

The problem is that I either spend too long making (unnecessarily) getting my "research" code to industry-quality, or I publish work based on "bad quality" code, and I feel like a fraud.

My career progression is dependent on me writing "bad" code!?

The "craft" of software development is a huge subject - but where is the best practice for academic research? Nobody writes unit tests for conference paper code!

Does anyone find them in a similar situation? Does anyone know of formal methodologies for "research" code?

share|improve this question
15  
4  
There's also a related question on SO: How can I write good “research code”? –  ff524 May 21 at 18:34
5  
Good code in academia is exactly the same thing as good code in industry. So, essentially, there are the same books, blogs, best practices and places to learn should work. In many places researchers are learning it, start using general-purpose languages and collaborate on code under VCS. But as long as they are getting paid and promoted for papers (even closed-access), not code (even open source), the code will be a tool not a priority (sadly). –  Piotr Migdal May 21 at 19:04
9  
@PiotrMigdal Good code does what is required. Not less, but certainly not more. If you are building a throw-away demonstrator for a conference using TDD with 100% test coverage, continuous integration, strict issue tracking and release management, you are over-engineering. Not all code needs to be maintainable, and many research prototypes certainly don't need to be. –  xLeitix May 21 at 19:24
4  
The Software Carpentry organisation may be of interest you - their aim is to train scientists on software development good practice (and also to encourage code-as-a-citable-object to improve reproducibility in science) software-carpentry.org –  Simon W May 26 at 10:12

8 Answers 8

I think the key to understanding research code for industrial software engineers is to accept that you are typically not building a product. You do not have customers as such. You are building software to prove a point.

As such, the majority of code that you write as a researcher is more akin to the throw-away prototypes and mockups that you (in industry) often write in the early phases of a project. As you are certainly aware, even in industry these mockups have quite different properties than the final software. They primarily need to:

  • Have exactly the features that you want to show to the customer. Not more, not less. Typically, all the boring standard features for the domain are omitted.
  • Need to get done quickly. Both you and the customer know that the prototype will be thrown away anyway, so it does not matter whether the code is maintainable.
  • Need to be easy to extend and adapt, optimally live during the demo.

Essentially the same properties are also useful for most throwaway research code. You do not want to build features that you do not need. You do not want to waste time writing e.g., maintainable code if you know that it will not be maintained. You want to use an environment that reduces the amount of boilerplate code and setup, and which maybe auto-generates a lot of code for you that is "good enough" for your demonstrator (Ruby on Rails and its scaffolding features come to mind).

My career progression is dependent on me writing "bad" code!?

No, it depends on you writing code fit for purpose. Just like in industry. In industry and academia nobody applauds you for software qualities that are not needed. Try to reconsider what the point of the code is that you are writing. If you plan to release your code as open source software and you expect it to be picked up by other people across the world, then go nuts - use all the engineering techniques you have also used in industry to build the best product you can. If your goal is to evaluate this one algorithm or principle for your conference, and then throw away the code, then you can also live happily without writing a single unit test without feeling like a fraud at all.

EDIT: of course this does not mean that it is acceptable to write code where you are unsure whether you have implemented said algorithm correctly. Ensuring that what you have indeed shown what you claim to have shown is mandatory, especially in research code.

share|improve this answer
3  
+1, especially for the edit. –  Angew May 22 at 11:19
    
For research code, it can be as sloppy and poorly-designed as you want, as long as you aren't going to use it later, release it for the public as open source, or maintain it in the future. But especially for research, I would say that unit tests are probably one of your best friends - because in the event that someone wants to check the validity of the code itself, actually working as intended (even if the research results are valid), then unit tests will be there to back you up. They're also useful for test-driven development, which I believe fits the field of research quite nicely. –  Chris Cirefice May 23 at 15:07
2  
Key phrase: "writing code fit for purpose". Excellent answer. –  Floris May 23 at 19:44

I either spend too long making (unnecessarily) getting my "research" code to industry-quality

Don't. Make it as good as possible within the suggested timeframe. Aiming for 100% perfection that will require double the time is not worth it. In this sense, research is exactly like industry.

Consequently, the "academic" code I've written is poor quality

That does not mean all academic code is of low quality. If you check papers on algorithms conferences or parallel processing systems, you will see that the developers have thought even excruciating details, like reordering of data for fewer cache misses, SIMD, GPU programming, SSD storage etc. Usually advanced CS algorithms research is some years ahead in adopting new methods, hardware techniques before any of those techniques actually hit the industry. On the other hand, in more theoretical CS conferences code is mainly a tool and as such, it does not have to be cutting edge. So, the quality is related to the audience of your product code (exactly like industry).

Does anyone know of formal methodologies for "research" code?

I have never heard on any methods especially tailored for research code. Still, you can use the practices from your industry background (when they actually accelerate your process of writing the software). For example a versioning system accelerates development and minimizes errors / losses of data. On the other hand UNIT tests take a lot of time, which you might not have. A informal wiki for bugs, documentation, features might worth the extra time, since it also accelerates writing the actual research paper. Contrarily, a full blown bug database (bugzilla) might not worth the extra time and effort.

So, stick to those industry methods, techniques you know will save you time on the long run and will improve your software but without taking all of your time. Finding a middle ground is always the best solution.

share|improve this answer
5  
On the other hand, in more theoretical CS conferences code is mainly a tool — More accurately, in more theoretical CS conferences, code doesn't exist at all. –  JeffE May 21 at 23:42
2  
"On the other hand UNIT tests take a lot of time, which you might not have." Not necessarily. In addition, time spent on formulating unit tests often helps with design (enforces modularity, creates a partial design spec), cuts down on time spent debugging, and can be used to help substantiate that research outputs are accurate. It may not be necessary to unit test everything, but for cases where it is not unduly difficult to unit test, the benefits are worth the investment. –  Geoff Oxberry May 22 at 9:34
    
@JeffE: Do you mean that computer code is not used, or simply that it is not discussed at conferences? I find the former implausible; for instance, someone studying geometric complexity theory might well use computer code to compute Littlewood-Richardson coefficients and other invariants related to representation theory. –  Charles Staats May 24 at 16:24
2  
I mean code is not even written, much less used. In 20+ years as a theoretical computer scientist, I have only written only one paper that required any code whatsoever. (Some of my colleagues think I should be ashamed by that clam, but I'm not.) –  JeffE May 24 at 16:34
    
Oh. Computer science, huh? How about psychologist writing SAS code? How about sociologists writing SPSS code (which is not even called code, it is referred to as "syntax" in SPSS)? How many papers in these conference would have good code? How many psychologists / sociologists would have but a single course in computer science? How many of them would know about revision control, let alone unit tests? This is Academia website, not computer science website. Please be more inclusive in your thinking. –  StasK Jun 14 at 4:05

tl;dr Some parts of industry "best practices" fit in well, and other parts are inefficient in a research environment. Keep what works well in this environment.


I'm a experimental particle physicist and we write a lot of code and much of it is big projects written by many, dispersed programmers.

Some part of the usual industry tool kit we use enthusiastically

  • Version control
  • Bug tracking
  • Automated build and test systems

Other part are either slower to catch on or not as highly prized, including

  • A formal Process (with a capital 'P') with regular planning meetings and release checklist and so on. These appear as projects get bigger (usually in response to a total break-down in quality control or long release lags). That is, we use them when we need them.

  • Documentation generation systems are pretty common but only lightly used until the project gets big when people who are forced to decode some bits often contribute a little more documentation.

  • Unit testing is sparse and usually concentrated on the lower layers of the systems, but regression testing is more common.

  • Most projects have coding standards, but they are generally loosely specified and weakly enforced.

Other things simple don't show up much.

  • Feature planning is pretty hard when you don't know what clever ideas a grad student will have next week to solve a problem that you haven't even noticed yet.

  • It used to be true that strict check-in control got in the way of spreading experimental code segments around, and we'd simply freeze new development check-ins occasionally to get out a blessed release (a situation that rendered HEAD/trunk/whatever a "use at your own risk" proposition). With the rise of distributed version control there is starting to be a stronger commitment to check-in controls for the official trunk.

  • Refactoring usually only happens when both a new person starts working with some old code and they feel they need changes or extensions. That which is not broken is left well enough alone.


From your question I suspect that you are doing your coding either single-handed or in a small group. In that setting the details change, but the tone remains the same. Simply keep the parts of your industry practice that work, and dispense with or delay the parts that have the worst cost/benefit ratio in terms of time/results.

Leave off the heavy refactoring until you know from evidence that a particular part of your code will be reused. Similarly, be content with rough documentation unless and until the code is shown to have a on-going life. And so on.

share|improve this answer
4  
Re: Formal review. One thing I notice is that in many groups, the researcher who is developing the software has much more experience and knowledge than the PI - so it is difficult for the PI to effectively monitor progress; whereas in the industry, there is always a senior programmer who has the appropriate skillset. –  Superbest May 22 at 1:46
    
Well, in a particle physics setting there are senior coding physicists with both the necessary skillsets. Admittedly there are usually only a few on each project, but they can be found. –  dmckee May 22 at 2:17

I am a researcher and self-taught developer. I have done substantial projects which were primarily software based. Although my work is far from the most "hardcore" stuff that's out there in terms of complexity and scale, the projects were big enough that naive mistakes (eg. not using version control or poorly documenting code) were very painful. I ended up learning quite a few "best practices" through trial and error.

I have also been on the receiving end of "unmaintanable code passed down to fellow researcher":

enter image description here

my industrial experience has been a hindrance in my research

No, you have basically come from a civilized environment that solved these problems decades ago into one that is stuck in the stone age in terms of software development hygiene. Scientists still code like it's the 60s. Of course you feel a conflict, but the fault is not with you.

In industry, code needs to be (ideally): maintainable, bug-free, refactored, well-documented, rigorously tested

Let's say the speaker at a scientific conference, while describing the computational part of his research, said one of the following:

"The code I wrote for this research is, admittedly..."

  • ...unmaintainable (and good luck building on my research!)
  • ...full of bugs (and I have no idea if the output is even correct!)
  • ...unreadable spagetti (and I don't even know how it works, let alone if it does so correctly!)
  • ...undocumented (and all the mistakes are obfuscated from reviewers!)
  • ...not tested (so god knows if it does what I say/think it does!)

Do you expect the audience to react with anything but scorn and outrage? If I heard such a thing, I would not believe anything this person published ever again.

In academia, the goal is (...) in the shortest possible time.

Yes, but "no shorter". You don't skip vital control experiments because "controls take time". You can't skimp on code quality for very similar reasons.

There seems to be no motivation to write [good code]

Because this is an endemic problem of academia. Although computers have been used in science for decades, it seems that algorithms have only become an important part of research in the last decade or so (perhaps because of "big data"). When you base your research on code, that code must be good quality. It is not enough to simply crank out some buggy write-only script and call it a day. The software development community has figured all of this out long ago, but academia has not yet caught on - I think the reason is that most scientists do not have a formal background in software development, and there have not been enough huge scandals in research caused by bad programming practice (eg. key results of a high-profile paper turn out to be artifacts caused by bugs).

Consider how, in many disciplines, reviewers will not even ask about the source code of your computation-heavy paper. How can they evaluate, then, the validity of your results? They cannot, and this is a failure of the peer review model as it currently exists.

Sorry to go on a rant, but basically, it's like this: As you know, there are very good reasons for writing quality code, even if no one is watching over your shoulder. In science, currently it so happens that nobody cares if your code is good or not. But this should not be a reason for you to not write good code anyway - the reasons for writing good code in the industry still largely apply to science.

Unfortunately, you may not be rewarded for your extra work. You may even be punished, because as you say, good code takes longer, and others may not see beyond that. Your PI or colleagues may not understand why you are so much slower. The best you can do is explain to them the need for good practices.

Obviously, there are exceptions. For instance, you may not need to worry about portability or backwards compatibility with old versions of the OS for code that is meant to run on a dedicated lab computer (although it is undesirable to write your code such that it only runs in a very exotic environment that other scientists will not be able to easily reconstruct). But by and large, I find that industry practices still apply, and the exceptions can be easily detected by applying a modicum of critical thought. That said, there is also a helpful publication called "Best Practices for Scientific Computing" which examines this matter in detail.

Ultimately, it is an ethical decision you must make. Do you care about doing good science above all else? Follow best practices. Do you want to cut corners that you shouldn't (in an ethical sense), to save time or avoid friction with co-workers? I couldn't recommend you to do this, on principle. But obviously many people do, and perhaps in practice, some scientists are forced to do it - although then again, does being unable to do good science by circumstance excuse bad science?

Also, like I said, I think part of the problem is that there haven't been any big scandals. If you do skimp on code quality, there's a chance it will catch up with you. You might even end up being one those big scandals. Admittedly, the risk is probably small... But, I think you can see my point.

share|improve this answer
1  
"If I heard such a thing, I would not believe anything this person published ever again" - hence, no one publishes really publishes code. –  Ben May 22 at 6:57
    
No formal tests doesn't mean I don't know if it works. Normally, when I write a function, I check it is correct, but don't take the time to write the full unittest structure. This means that I am fairly sure that it will work until I refactor it. Also, throwing several asserts in the code helps with correctness. –  Davidmh May 22 at 17:45
1  
@Davidmh You are correct that an untested function is not necessarily incorrect. But I think tests are the proof that it is. –  Superbest May 23 at 4:04
1  
"Consider how, in many disciplines, reviewers will not even ask about the source code of your computation-heavy paper. How can they evaluate, then, the validity of your results? They cannot, and this is a failure of the peer review model as it currently exists." This is a problem for all academic research -- the researcher advances his career by making a point, and his methods are rarely double-checked (almost never before the next grant or tenure decision). There is huge incentive to cut corners and fill in the gaps with bullshit. –  adam.r May 25 at 21:41
1  
@adam.r But there is a big difference. For benchtop experiments, you are expected to specify in great detail where the reagents were obtained and how the experiments were made, to the point that someone could repeat the experiment by reading your paper. Sure, some reviewers don't do a good job, and as a consequence there are papers with very vague methods sections, but writing up methodology accepted as a responsibility of the reviewer/author. But publishing or reviewing source code is not treated as such a responsibility in many cases. –  Superbest May 25 at 21:44

The main characteristic of research code (more so than typical programs) is that it is harder to plan. Research by definition reaches into the unknown, and this also translates in program structures. As a researcher you often don't have the time to refactor when your research slithers into a different direction that should translate into a different software structure.

In typical software engineering you (should) have a fairly good idea of the final functionality before writing a single line of code. In research, this is often not the case. Programming for research purposes is mainly about rapid prototyping, which is typically done differently than programming for long term use (e.g. little to no unit tests, use of different languages, ...). The main mantra is to get results fast, not optimal from a software engineering perspective.

Finally, proper software engineering is a fine art that is very hard to master. Even in industry settings, ugly software is abundant (when written by professionals!). The average researcher has no formal training in software writing.

My career progression is dependent on me writing "bad" code!?

As a researcher, you are not paid to develop software (unfortunately). As an idealist I like to believe that this will change over time, but for now funding sources only care about papers.

The "craft" of software development is a huge subject - but where is the best practice for academic research? Nobody writes unit tests for conference paper code!

Unit tests serve two main purposes: (i) assert that code is working correctly and (ii) find and debug problems fast, particularly after structural changes and refactoring (long term benefit for large infrastructures). As research software is typically fairly small, the first advantage is the only one that is really relevant. It seems that this advantage is either too small or is being underestimated (again, recall that most researchers are not software engineers).

Does anyone find them in a similar situation? Does anyone know of formal methodologies for "research" code?

If you want to change the world, start with yourself. I personally make a point of providing software along with a manuscript whenever it is reasonable. I also consistently ask for code as a reviewer, though this seems uncommon.

share|improve this answer
    
This seems like an answer to Why do many talented scientists write horrible software?, not to the current question on best practices. –  ff524 May 21 at 18:28
    
@ff524 work in progress :) –  Marc Claesen May 21 at 18:32

I think your problem is in seeing quality as binary.

Whether in industry or academia we need to make choices as to particular quality goals. An ‘industrial’ piece of software might be safety critical and demand the highest achievable level of quality, or it might be an internally used development tool.

Quality goals might be:

  • Uptime
    • Probably not a concern for you
  • Robustness
    • Will you need to run with a lot of different data sets, or is it a one off?
  • Re-usability
    • Will the processing be incorporated into the next problem you work on?
  • Ability to verify
    • Does it work the way is it is supposed to? What would it mean if you published and this was not true?
  • Ability to validate
    • Does it produce the right answer? [ Hopefully, this does matter. ]
    • Another way of checking the correctness of the answer might mean it does not matter if the logic getting there is wrong.
  • Needs to be maintainable by
    • Self
    • Other expert
    • Some random undergraduate
    • Or definitely is throw away.
  • Suitable to be made public for use by others to check or re-use your work
  • Etc.

Having articulated your quality goals, your development and build system should be sufficient to meet those goals and no more. Note that this might vary from project to project.

share|improve this answer

Rather than talking about differences with industry software development (which I have no experience with), I'd like to talk about things you should do in academia. I will be assuming that your code is not there for its own sake but attaches to some scientific statement such as "E. coli shares the foo-genome with humans" or "Algorithm A outperforms algorithm B in scenario Z".

  • Best Effort

    While your goal is not productive use with revenue attached to it, you should have a reasonable amount of certainty that your code does what you claim and can be understood (peer reviewed!) by interested parties. That is, write clear code, comment and document. And (unit-)test.

    If nothing else, remember that you may have to revisit your own code some time later. You build a protoype but your next one may reuse parts of it. Or you want students to extend upon it.

  • Accessibility

    In order to support scientific evalutation of your work (falsifiability, reproducability) other researchers have to have to be able to compile and execute your programs.

    Therefore, you should provide sources, build files/instructions and whatever input data you used in your work. Keep in mind that someone may want to build your program years after initial publishing, so make sure that the sources are still around then and the build process/instructions are reasonably robust against time (mention library versions). UI is not too important but "best effort" applies here, too.

    Consider putting your code on Github or a similar platform (e.g. your own). That way, you can publish updates and collect bugs easily. See also here and here.

  • Licensing

    You should say something about what others (in particular fellow researchers) may or may not do with your code. You can use any license (I'd argue it should allow at least the liberties of GPL). The CRAPL might be worth a look.

  • Fair Evaluation

    If your algorithm/code is the artifact you propose (as superior) you have to compare it to existing solutions. Make sure to use comparable input, rerun the experiment for the alternatives and follow basic experimental best practice (check out McGeoch's work, for instance). Make sure your comparison includes the accepted standard(s) of your field if there are any; if your approach yields different results, you have to explain why (that's okay/correct/better).

Accessibility is probably the most crucial item. Research code needs to be shared. I think this can't be stressed enough, and it's one of the major problems in all of computational science. Other than, say, physics, we have to opportunity to easily share and reproduce (most) experimental setup -- we have to make use of that.

Personally, I don't see why any article that founds its claims on computations that reviewers (and other parties) can not reproduce has any right to be published.


One reminder for all theory types:

Beware of bugs in the above code; I have only proved it correct, not tried it.
Donald E. Knuth

share|improve this answer
    
"Best effort" meant that if you don't have programming expertise to speak of, don't do it. Hire a person to do it. –  Raphael May 22 at 10:01
    
I hope you aren't seriously recommending the CRAPL license, which is a farce. Example: "By reading this sentence, You have agreed to the terms and conditions of this License." If licenses worked that way we'd all be in deep trouble! –  wjl May 25 at 14:49
    
@wjl: I think it's worth a look, as I've stated. It contains some interesting thoughts, imho. –  Raphael May 25 at 19:25
1  
This is the only answer that talked about reproducibility. Isn't that supposed to be cornerstone of science? If your code does not work, or cannot be used otherwise, your so-called "research" is not reproducible, and should not have been published in first place. –  StasK Jun 14 at 4:15

I was directed to this question by a chap in answer to my request for clarification "Why they were using a float type as a key to hash". Now to me this did not seem like a good idea. After a bit of back and forth, the questioner directed me here as their reason for doing it.

That was interesting as is the question itself.

Are you persuaded to write bad code in order to fulfill certain academic goals? Yes, even in CS courses I might add. At least bad in terms of comprehensibility. However as we are all too horribly aware, we have been 'persuaded' to write equally bad code in the commercial environment, or our lords and masters have been persuaded to have poor code written for them....

Leaving aside trivial implementations of algorithms. For instance should you use i and j as loop variables in a bubble sort algorithm, or indeed should you use a bubble sort at all. My response to your question, would be another one. How do do good software engineering principles help you achieve your goals?

Could, say good naming, SOLID principles, coherence and coupling et al get you to your goal more efficiently? My answer would be almost certainly. They are designed to do that in any non-trivial piece of software. It's what they are for, they were created in order to achieve software that could be changed at less cost. It doesn't matter that you aren't (or at least cannot foresee) a version two, the implementation isn't springing full blown from your mind, so it will be changing.

If you aren't sure what code you need to write, then something like TDD would also help given you have a ready made unit testing environment. Even without that "luxury" writing testable code is going to.

You have a huge advantage over your fellows who don't have a software engineering background, you should be able to get that irritating binary bit of the exercise out of the way quicker, get to the meat of your goal with less effort and then be able to expend more effort on the real goal.

I once had a discussion with an academically qualified type who told me re-factoring was just software aesthetes messing about and that I should have written the software correctly in the first place. Needless to say my respect for this individual dropped a notch or two, which was unfortunate for him as he hadn't earned that much in the first place.

So in summation I would say the sensible option would be to use good software engineering practices in all your efforts. Just to be clear though it is not good software engineering practice to write good code that you do not need...

To paraphrase Gandalf, "Keep it simple, keep it safe"

As for should you use a float as a key to hash to save time. If you know without any doubt as you choose that design compromise that the problems with doing so will not cause you an issue, then perhaps. But the amount of effort required to evade that compromise is fairly trivial and you just spent time asking how to get round an issue with using a float as a key. I rest my case

In any environment commercial or academic, choosing to lower quality, is both a benefit and a cost, examine both....

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.