Take the 2-minute tour ×
Academia Stack Exchange is a question and answer site for academics and those enrolled in higher education. It's 100% free, no registration required.

I'm studying published journal articles on my topic, and I'd like to reproduce their analysis. How can I get a copy of their dataset and enough specifications to reproduce their numerical results?

In my specific case, I'm studying Stecklov et al. (2005), published in Demography, and they're using PROGRESA datasets which are publicly available, but when I try to reproduce their results, all the base statistics are off by a bit, and the regressions come out a bit different. Probably I'm just including or excluding a few different cases, but what can I do to identify the difference?

share|improve this question
12  
I'm not sure how we can help. The authors of the paper are in a better position to help, and most authors will respond to a polite request. It's likely that there were some undocumented steps in data handling. –  Jason B Oct 24 '11 at 12:59
1  
Get into contact with the authors so that they can share their secrets with you. As Jason said, the fact that you cannot replicate the results is probably due to a poor documentation of the data handling. I have faced a similar situation some years ago and I have learned that reproducible research is for the others ... Good luck! –  user357 Nov 14 '11 at 16:30
1  
I was trying to replicate a paper of my professor by implementing the algorithm he mentioned in his paper. I proved that his algorithm written on the paper was wrong, and he accepted it too as a typo. Sometimes people love waiving their hands. I don't think any reviewer sits down and replicates the paper. –  user2792 Oct 6 '12 at 14:58
    
You could also take a look at this related question which has several useful answers about requesting data from published research. –  Jez Dec 11 '12 at 12:16
add comment

migrated from economics.stackexchange.com May 3 '12 at 18:11

This question came from our site for economists and graduate-level economics students.

3 Answers

Email the author nominated for communications, and ask them for a copy of their dataset and enough specifications to reproduce their numeric results. As a backstop, ask for the specific criteria they used to exclude cases. Consider offering co-authorship on any papers that will result from your analysis, as incentive. Give them enough context about you & your own work, so that they know your enquiry is serious, and that you're looking to constructively build on their work, (and not to stitch them up!)

share|improve this answer
11  
Co-authorship seems a bit extreme, and possibly unethical. –  Ari B. Friedman May 4 '12 at 11:34
add comment

Read carefully, underline everything it says about anything to do with the data, but it is likely you won't get it quite right. For Labor we were assigned to reproduce "Tax Incentives and the Decision to Purchase Health Insurance: Evidence from the Self-Employed" by Jonathan Gruber and James Poterba (QJE, 1994) and "Children and Their Parents' Labor Supply: Evidence from Exogenous Variation in Family Size" by Joshua D. Angrist and William N. Evans (AER, 1998). Try as we might, no one in the class ever got exactly the same things as in the papers. It was always about right, and enough people did it independently that we trusted we were doing the right thing. After a while the small differences stopped bothering my classmates and I.

share|improve this answer
    
Out of curiosity, what class was that where you replicated papers? –  dchandler Oct 24 '11 at 4:53
    
The first class of a PhD labor sequence. –  Bitsy Oct 24 '11 at 14:50
add comment

Replication is an incredibly important part of academia. Don't forget that it's possible you are failing to replicate because the original authors did something wrong! You should absolutely email them, but keep in mind that failure to replicate can be publishable, provided your differences are significant and you can couch couch our results in current theory.

Have a look at Gary King's "Replication, Replication"

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.