6

I am a first-year PhD student, and I am trying to learn how researchers write papers and reproduce existing work.

One thing I’ve noticed is that, for instance, paper X presents results from dataset D. However, two other papers, Y and Z, which use the same dataset D, cite paper X and compare their results with it. Interestingly, the numbers that papers Y and Z report for paper X's results are slightly different from the original results in paper X and even from each other.

Is this happening because the authors of papers Y and Z reproduced the results from paper X and used those for comparison? What happens if the reproduced results of paper X are lower than the published results?

How can we trust the reproduced results of paper X from the authors of papers Y and Z when comparing them? Could it be that they used slightly different data preprocessing techniques that lowered the results for paper X but increased their own results?

Improve this question
2
  • 1
    Do Y and Z state why they present results different from what they cite? Is the difference relevant for the gist of their comparison?
    – MisterMiyagi
    Commented Oct 13 at 6:11
  • @MisterMiyagi No, they just put numbers in the table, and don't talk about it.
    – Park Bo
    Commented Oct 13 at 15:41

1 Answer 1

Reset to default
10

Medical PhD student here. In my Bachelor's Thesis, I used a publicly available dataset in order to test some hypothesis, different from the hypothesis the original authors tested. Before doing my analysis, I first tried to replicate the analysis of the original authors. To my surprise, although I could compute the same values for means and counts, I couldn't replicate their values for multivariate models. In the original article, the authors explained that they "controlled for factors such as age, gender and familial history", providing HRs and ORs only for the things they were interested in, but not for any of the cofounders.

What might be happening is that the original authors have not provided enough information about how they analyzed the data, thus making it very hard for others to "guess" the exact methods/models they used.

Moreover, different software (and different versions of the same software) can use slightly different methods/models/statistical tests. SPSS is a good example, newer versions using Lilliefors test instead of Kolmogorov-Smirnov for assessing normality. R also can use different types for sum of squares in ANOVA, while SPSS defaults to Type III (I think), without letting you choose. https://stats.stackexchange.com/questions/20452/how-to-interpret-type-i-type-ii-and-type-iii-anova-and-manova

In multivariate models, depending on the software used, different results can be obtained by simply choosing a different order for the predictors, or using a stepwise method instead of the enter method.

In genetics, depending on the protocols/machines used for reading DNA/RNA sequences, raw sequence data might contain "barcode" sequences that would be automatically discarded by the (proprietary) software used for data cleanup after reading, but you may not have this exact information or access to the cleanup software, while the repository requires authors to upload raw readings before cleanup. Different reference genomes might have been used (see HG19 vs HG38). https://www.thebiomics.com/research/human-genome-assembly/hg19-vs-hg38#:~:text=HG38%20includes%20additional%20data%20that,version%20of%20the%20human%20genome%20.

A personal beef of mine is with Brainstorm, which computes quantitative electroencephalography values based on EEG recordings. They don't provide an archive of past versions, and this year's versions have made significant changes to the interface, not only changing defaults but completely removing some options that I used last year in computing values. Essentially, I'm unable to reproduce past results using the only version available today. https://neuroimage.usc.edu/brainstorm/Introduction

Improve this answer
4
  • 1
    The tl;dr version of this nice answer is that what the OP has observed happens often for a variety of reasons.
    – Ethan Bolker
    Commented Oct 13 at 14:53
  • Thank you for depth explanation! Yes, I wonder later if I were to write paper that needs to compare with others, should I use number from paper or number I got from reproducing their work, which is slightly lower than original work.
    – Park Bo
    Commented Oct 13 at 15:43
  • 1
    @ParkBo I'd recommend using the number from your reproduction, so you are making an accurate comparison, such as the exact same statistical test. If the difference between your reproduction and their paper is more than a rounding error, though, I would add both rows to the table.
    – Davidmh
    Commented Oct 13 at 17:51
  • @Davidmh Thank you
    – Park Bo
    Commented Oct 13 at 19:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .