A study compared two groups (differing on factor A) on ability X and found that X differs between the two groups (effect of A). However, the groups also differ on a not interesting variable B (say, "age"). When using B as covariate in the analysis of A on X, there is no effect of A.

In the statistical analysis (ANCOVA) the authors present the statistics for X corrected for B (i.e. "no significant effect of A on X"), however, next to this analysis, they present the effect sizes (Cohen's d) of the raw/uncorrected values of X as support for their hypothesis ("large effect of A on X"). They do not attempt to make it obvious that these effect sizes are based on scores without correcting for B.

I think they have to present the effect sizes of A corrected for B (i.e. after regressing out the effect of B on X). However, they don't want to, possibly because they have a strong hypothesis about A affecting X.

Is what they are doing correct, is it normal, is it misleading or even fraud?

  • Might this question be a better fit for Statistics SE? – astronat Sep 25 '17 at 8:19

You found out, so apparently there is enough information present in the article to see what they were doing. So that in my mind rules out fraud.

Whether or not it is correct, normal, or misleading depends on the finer details. There is also another possibility: it is simply a mistake. Researchers are human after all.

  • 1
    I found out through communication with the authors, the information is not present in the paper. There I also explained what I though would be correct (i.e. presenting effect sizes based on data corrected for B), upon which they responded that what they are doing is what they want to do, thus excluding human mistake. – Mark Sep 25 '17 at 8:36
  • The fact that they want to report this does not exclude human mistake. Statistics is a necessary evil for most substantive researchers, so it is common that they are not as up to date as one would have liked. – Maarten Buis Sep 25 '17 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.