Like most fields that rely on statistical analyses, economics has suffered from a few well-publicized coding errors (most notably the Foote and Goetz finding that when correcting Donohue and Levitt's programming error in the abortion/crime paper the conclusion is reversed), and likely suffers from far more which are never discovered.

What solutions have other fields used to ameliorate this problem, and how might the incentives of researchers be changed to encourage them to submit to these changes?

  • 1
    I give a related answer here. – Andy W Dec 18 '13 at 20:10

You need the journals...

Nothing will move without it. The American Economics Review has taken the lead in requiring all data papers to make their data and source code available. Unfortunately, there's no real indication that other journals will follow suit, despite the formidable reputation of AER. Sadly, even the AER doesn't have a clear repository and not all code is available even though they require it of the authors.

Beyond that, David Card has a nice repository of sorts for structural econometrics data. Josh Angrist and David Autor should be praised for creating Data Archives that document their own work. But at this point it's still up to individuals to make their research transparent and their code available.

For what it's worth, I've been thinking about this issue a lot lately and decided to create a Google Code Project where economists can upload their code: http://code.google.com/p/econ-code/ ... That said, I have not yet tried to publicize it and think the ultimate key to adoption lies with the journals.

  • I think the journals are a big part of the picture. Individual examples of code transparency abound (Raj Chetty publishes everything on his website), but until there's a) a structure for archiving things, and b) a requirement to do so, I doubt it will achieve mass acceptance. It's interesting that AER hasn't been followed in its archiving requirement. When NEJM and a few other top journals got together and mandated registering clinical trials in advance, the other journals mostly fell into line.... – gsk3 Oct 16 '11 at 11:47
  • Publishing code is one thing, but what about the data? If you are using publicly available data, then there should be no excuse for not being transparent. But what about those who are working with confidential datasets? – user357 Nov 14 '11 at 16:39
  • For publicly available data, transparency would dictate that code should go from the "raw data" that is publicly available to the fully transformed data. If not available via internet, it should be posted. The one "downside" to having to make data and data transformations public is that it lowers the reward to cleverly compiling/cleaning data sets if you don't have a monopoly on using them. As for confidentiality, researchers must respect confidential or proprietary data. However, in that case they should provide "example data" so that you can understand how the code worked. – dchandler Nov 15 '11 at 19:34
  • Journals are not magical entities - they are run by human beings, usually prominent economists. So the question is why these economists haven't pushed for change. I think partly it is that these economists are precisely the ones who have succeeded under the status quo, so they have little incentive to change the system. If they succeeded in the past with possibly sloppy and erroneous work for which they were not required to post their data and code, they will likely to continue prospering under the status quo. – Kenny LJ Aug 3 '14 at 4:13

Warning - anecdotal evidence ahead:

We have a couple of pet statisticians that we run things past: they review our statistical methodology, and can check that the code does what we think it's doing. (That is to say, we borrow a few hours of time from colleagues in other departments. And in some funding bids / project proposals, we explicitly put in time for them). In some cases, we've coded up algorithms in different languages, and checked that results have been reproduced.

The incentives for cross-disciplinary collaboration are, I believe, already there. When we've explained to our statisticians what we're trying to do, for a stats health check, they've often been able to suggest additional tests. And they love getting their paws on new datasets, to go mining on. So it's constructive for all parties.

  • 5
    But to me internal-collaboration is not the same thing as code transparency. "Many eyes make all bugs shallow," relies on n >> the size of a small team. – gsk3 Oct 16 '11 at 11:47

Another approach is to ease the process of making your data/code open. The Center for Open Science is an interesting non-profit that started this year. They're developing The Open Science Framework, which is a tool meant to assist with the research workflow--it facilitates collaboration, version control, and it reduces making your data/code open (completely or in parts) to a single click. I guess the strategy is to lure researchers into using the software by making a useful tool, and hoping that if openness is only a click away, more of them will just click the button. I could see this or something like it making a real impact.

  • I like this approach, but it will take a long time for the tools that researchers are already using to incorporate it. – Ari B. Friedman Dec 2 '13 at 17:22

I agree with dchandler. The journals have to require it and publicize it on their websites. There is no other way. There is another example from S. D. Levitt of Freakonomics fame in which a paper had erroneous conclusions that were demonstrated by trying to replicate the results. This is a serious issue that needs attention. Scientific rigor goes as far as the academic rigor, but academics have regrettably, large incentives to "make mistakes" in their coding to obtain conclusions beneficial to their research programme. There hardly is a more pressing issue in academic research than making the research process public. Technically it is a breeze to do, what is required is the political will for the journals to implement it. Hopefully, the issue can be raised in future annual meetings of the corresponding associations.

  • Not necessarily "a breeze," particularly the way most people code. But point taken. – Ari B. Friedman Dec 18 '13 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.