As the title says.

My background is in Economics/Finance(mostly), many topics in those fields (and I am sure other fields) require fairly complicated programming, enough to where one can easily screw something up. How do Academic journals defend against results that are generated by bugs?

As far as I understand nobody ever sees my code, it could be hundreds or even thousands lines of garbage code without a single function in it (not to mention no unit tests) laced with bugs, and I kept "fixing" things until my results "made sense", and then happily reported them. How could a journal tell that my results are trash?

  • 46
    Short answer: They don't.
    – JeffE
    Mar 14, 2014 at 1:05
  • 3
    I think it will be interesting when more people share their code and data. Finding little bugs may become more routine, just as it is with non-academic coding projects. Mar 14, 2014 at 7:25
  • 8
    This is a very good point for making the code publicly available. I have seen publications on data analysed with software that is not only not available, but the whole algorithm is secret even for the researchers that did the analysis.
    – Davidmh
    Mar 14, 2014 at 10:30
  • 2
    @Davidmh that strangely reminds me of this SMBC comic. Mar 14, 2014 at 12:25
  • 2
    IMO, if journals gave preference to papers that come a long with a link to github, that contains clean readable code, even better with unit tests, it would help bug issue quite a bit. Others could find bugs, plus original authors would be forced to write clean code, that would help them from making mistakes. Also, it is often not clear what exactly certain steps in a paper are doing, having access to the source could would help understand every detail of the paper.
    – Akavall
    Mar 14, 2014 at 15:35

3 Answers 3


Journals never make any guarantees regarding the validity of the content published in them, though this may seem to be implied. Ideally, errors are caught during the review process. Note that this problem is not specific to programming bugs, subtle errors can occur in all kinds of settings including physical experiments. It is important to always be critical about any results in any article, even highly cited articles in top journals (though these are less likely to be wrong, they still might be).

Letters to the editor are not uncommon in my domain when questionable results are published. In a worst case scenario, published papers can get retracted after the validity of their results has been formally rejected. Retractions for this reason seem to be fairly rare, though.

This is one of the reasons why reproducible results are so important. If several independent researchers seem to reach similar conclusions, they are likely correct.

  • 1
    Thanks for the answer. You say that if several independent researchers reach similar conclusions than the results are likely to be correct. However, if you publish some results, and than I find the same thing, would journals publish my paper? It has value because it supports your result, but on the other hand I didn't do anything new, so my gut feeling is that my paper won't be published.
    – Akavall
    Mar 14, 2014 at 1:49
  • 12
    @Akavall Presumably once you verify the original results, you can add/extend/apply the code to another problem. This makes your new results slightly more trustworthy since you've shown you can match what others have done. So the new work is why a paper gets published, and verifying the other results is part of your justification for the new work.
    – tpg2114
    Mar 14, 2014 at 3:33
  • 2
    "If several independent researchers seem to reach similar conclusions, they are likely correct"... UNLESS they have all used the same code :-)
    – Flyto
    Mar 20, 2014 at 11:11
  • @Flyto You are absolutely right. That's what I meant by independent. Mar 20, 2014 at 12:12
  • heh, fair point. But at least in some fields, I would not be surprised to see researchers considering themselves independant of one another while using the same written-by-others tools.
    – Flyto
    Mar 20, 2014 at 14:19

In addition to Marc Claesen's answer:

  • There exist a few journals that require the code to be submitted and where the review explicitly included that code, e.g. the Journal of Statistical Software

  • In one of my last papers, one reviewer asked exactly this question. Here's our reply in the text:

    The checks include unit tests to ensure calculational correctness, which consist of ca. twice as many lines of code than the actual function definitions.

    I include this because even if the journal does not have an explicit policy regarding the code, both reviewers and authors can already start with better coding and testing practices: I'm encouraged by this experience to include such statements also in future as author, and I will ask as reviewer.


This is a problem that is beginning to be recognised, and has been described by some as a "crisis of reproducibility". There have been examples of papers in prominent journals being withdrawn after bugs were found in the researchers' code. This article describes some of the problems in more detail.

In my view there are three primary paths to addressing it,

  1. Teach scientific programmers good software development practice
  2. Make source code and datasets available and citable, with DOIs and with confidence that they will be available and unchanged for the long term.
  3. Get journals to require that source and data are available (although comprimises must probably be made where data is commercially confidential), and that reviewers conduct code review. This may not be straightforward since (as noted in the Post & Votta article linked above) it will not always be possible to find one reviewer who is qualified to review the code and sufficiently expert on the science involved. (and that's before we even consider how long code review might take!)

The Software Carpentry project is aimed at addressing point 1 and 2 above, and may be of interest.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .