Statistical precedent?

Question

Suppose that an author [or team] writes a paper making heavy yet very questionable use of some particular statistics, without a convincing argument, demonstration, or test that the statistics are appropriate for how (and on what data) they are being used.

The argument in support of their use relies primarily on citations to previously published work in the same venue (generally regarded as a high-quality venue) where the new work is being submitted. While statistics are often important in that field, the reviewers are not necessarily statistics experts and may not have given those previous papers any detailed statistical review.

What impact, if any, should these prior publications have on the review of the new work?

Answer 1

I have been in that situation as a statistician moonlighting in psychology. I have seen my fair share of voodoo statistics here. (Sadly enough, the original manuscript title did not make it into the final article.)

My opinion is that prior errors do not justify later errors. I call out statistical errors and insist they be fixed, no matter how many authors have made the same error before.

Yes, I see the point that then new results cannot be compared to previous ones. But then, how reliable or useful were those previous results, anyway? Standing on the shoulders of giants is not overly useful if those giants are walking in the wrong direction.

It is always useful to refer to published work on specific errors if this exists - especially published work in venues that cater to the specific field. Psychologists will trust a statistical exposition more if it comes from a psychological journal, rather than JASA. I have lost count of how often I have referred to Miller & Chapman's "Misunderstanding Analysis of Covariance" (2001). Statistics in Medicine also often has articles that explain particular misconceptions. If there is nothing in article form, I sometimes include links to CrossValidated.