Suppose that an author [or team] writes a paper making heavy yet very questionable use of some particular statistics, without a convincing argument, demonstration, or test that the statistics are appropriate for how (and on what data) they are being used.

The argument in support of their use relies primarily on citations to previously published work in the same venue (generally regarded as a high-quality venue) where the new work is being submitted. While statistics are often important in that field, the reviewers are not necessarily statistics experts and may not have given those previous papers any detailed statistical review.

What impact, if any, should these prior publications have on the review of the new work?


I have been in that situation as a statistician moonlighting in psychology. I have seen my fair share of voodoo statistics here. (Sadly enough, the original manuscript title did not make it into the final article.)

My opinion is that prior errors do not justify later errors. I call out statistical errors and insist they be fixed, no matter how many authors have made the same error before.

Yes, I see the point that then new results cannot be compared to previous ones. But then, how reliable or useful were those previous results, anyway? Standing on the shoulders of giants is not overly useful if those giants are walking in the wrong direction.

It is always useful to refer to published work on specific errors if this exists - especially published work in venues that cater to the specific field. Psychologists will trust a statistical exposition more if it comes from a psychological journal, rather than JASA. I have lost count of how often I have referred to Miller & Chapman's "Misunderstanding Analysis of Covariance" (2001). Statistics in Medicine also often has articles that explain particular misconceptions. If there is nothing in article form, I sometimes include links to CrossValidated.

| improve this answer | |
  • That original titel would have looked nice on your résumé... On topic, your point on the shoulders of giants is spot on IMHO... Sadly it's often overlooked is many applied fields – Repmat Nov 29 '16 at 8:08
  • 1
    @Repmat: just to avoid confusion - the voodoo paper is not mine. Its (original) title is just extremely spot on. – Stephan Kolassa Nov 29 '16 at 8:18
  • Well oops, I just assumed... – Repmat Nov 29 '16 at 8:19
  • "Sadly enough, the original manuscript title did not make it into the final article.)" Yes it did -- not just as the title. (It seems slightly odd to me to have a footnote proclaiming your former title.) – Pete L. Clark Nov 30 '16 at 6:24
  • 1
    @PeteL.Clark: I don't know whether it was the case for this paper, but if a paper was widely distributed, and possible cited, as a preprint with one title, and the journal insisted on changing the title, then such a footnote could prevent some confusion. – Mark Meckes Dec 1 '16 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.