Tell me more ×
Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It's 100% free, no registration required.
up vote 16 down vote favorite
2

Like most fields that rely on statistical analyses, economics has suffered from a few well-publicized coding errors (most notably the Foote and Goetz finding that when correcting Donohue and Levitt's programming error in the abortion/crime paper the conclusion is reversed), and likely suffers from far more which are never discovered.

What solutions have other fields used to ameliorate this problem, and how might the incentives of researchers be changed to encourage them to submit to these changes?

share|improve this question
@johnny care to elaborate? – gsk3 Oct 18 '11 at 21:16
@dchandler Care to make that an answer? It deserves +many for doing something about the problem :-) – gsk3 Oct 24 '11 at 15:44
added it to my answer. thanks for suggesting I add it! – dchandler Oct 24 '11 at 17:37

migrated from economics.stackexchange.com May 2 '12 at 20:39

2 Answers

up vote 13 down vote accepted

You need the journals...

Nothing will move without it. The American Economics Review has taken the lead in requiring all data papers to make their data and source code available. Unfortunately, there's no real indication that other journals will follow suit, despite the formidable reputation of AER. Sadly, even the AER doesn't have a clear repository and not all code is available even though they require it of the authors.

Beyond that, David Card has a nice repository of sorts for structural econometrics data. Josh Angrist and David Autor should be praised for creating Data Archives that document their own work. But at this point it's still up to individuals to make their research transparent and their code available.

For what it's worth, I've been thinking about this issue a lot lately and decided to create a Google Code Project where economists can upload their code: http://code.google.com/p/econ-code/ ... That said, I have not yet tried to publicize it and think the ultimate key to adoption lies with the journals.

share|improve this answer
I think the journals are a big part of the picture. Individual examples of code transparency abound (Raj Chetty publishes everything on his website), but until there's a) a structure for archiving things, and b) a requirement to do so, I doubt it will achieve mass acceptance. It's interesting that AER hasn't been followed in its archiving requirement. When NEJM and a few other top journals got together and mandated registering clinical trials in advance, the other journals mostly fell into line.... – gsk3 Oct 16 '11 at 11:47
Publishing code is one thing, but what about the data? If you are using publicly available data, then there should be no excuse for not being transparent. But what about those who are working with confidential datasets? – user357 Nov 14 '11 at 16:39
For publicly available data, transparency would dictate that code should go from the "raw data" that is publicly available to the fully transformed data. If not available via internet, it should be posted. The one "downside" to having to make data and data transformations public is that it lowers the reward to cleverly compiling/cleaning data sets if you don't have a monopoly on using them. As for confidentiality, researchers must respect confidential or proprietary data. However, in that case they should provide "example data" so that you can understand how the code worked. – dchandler Nov 15 '11 at 19:34
up vote 6 down vote

Warning - anecdotal evidence ahead:

We have a couple of pet statisticians that we run things past: they review our statistical methodology, and can check that the code does what we think it's doing. (That is to say, we borrow a few hours of time from colleagues in other departments. And in some funding bids / project proposals, we explicitly put in time for them). In some cases, we've coded up algorithms in different languages, and checked that results have been reproduced.

The incentives for cross-disciplinary collaboration are, I believe, already there. When we've explained to our statisticians what we're trying to do, for a stats health check, they've often been able to suggest additional tests. And they love getting their paws on new datasets, to go mining on. So it's constructive for all parties.

share|improve this answer
5  
But to me internal-collaboration is not the same thing as code transparency. "Many eyes make all bugs shallow," relies on n >> the size of a small team. – gsk3 Oct 16 '11 at 11:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.