18

I was asked to serve as a reviewer in a peer-review process. I remarked some issues and suggested a minor revision to the editor. Now I received the revision back and generally think the authors did a good job revising the manuscript. The field is psychology.

There is one issue that I was not sure about. That is, the authors changed their hypotheses between drafts. This has something to do with the way they treated their data. In the first draft, the experiment provided data from two instruments (very similar instruments) and were analysed separately. Hypotheses were formulated separately for the two instruments.

Now, in the 2nd draft the instruments were combined and the data treated as coming from one instrument. Hypotheses were adapted to these new circumstances, but also were partly contradictory to previous hypotheses because then, the authors expected some differential results from the two instruments.

I am not sure how to judge this. Is this a big issue or just a minor thing? Why is it an issue at all? I have a feeling that this not too uncommon in my field (psychology), I mean to treat hypothesis testing somewhat lax. I believe that in grad school I learned that hypotheses have to be stated before conducting the experiment and cannot change. But I was googling a little about this topic and did not find anything that would indicate that post-hoc hypotheses are a bad thing.

2
  • 1
    When you were taught that post-hoc hypothesis is a bad thing, they did explain why it is bad? It essentially allows to tailor results to match conclusion. So, the question is really if the new version manages to convince you that no such misconduct (willful or accidental) has taken place. (Another issue is if the narrative is reliable in other respects, which with such big rewriting of the history is doubtful.)
    – Boris Bukh
    Mar 18, 2016 at 10:53
  • 6
    It is a big issue. I'd refer you to Simmons et al.'s excellent paper, "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant." pss.sagepub.com/content/early/2011/10/17/…
    – half-pass
    Mar 18, 2016 at 11:32

3 Answers 3

13

There is substantial divergence between fields and individuals to what extend twiddling with the original hypotheses post-hoc should be tolerated. Defensible positions range from "it's not a big deal if the experiment / write-up still makes sense with the new hypothesis" to "hypotheses shall never be changed". Only you can tell what your and your field's methodological stance on this is.

As Boris says, the main issue to avoid is "p-value hacking". That is, it should not be that the authors decide that whatever they find support in their data is redefined to have been their hypothesis all along - while exploratory research per se is not bad, it is distinctly different from hypothesis-driven research and should methodologically not be sold as such. You will need to evaluate whether this has happened for your manuscript. If the answer is "yes", this is certainly grounds for rejection.

However, from your explanation, it sounds to me like the authors did more of a technical refinement of their hypothesis, without changing the nature of it. If that is true, it sounds somewhat nit-picky to reject a paper on this grounds. I tend to be of the opinion that methodology should be considered a means to an end, not a strict rulebook that needs to be followed to the letter even if there appears to be no reason for some of its details in a specific case.

2
  • I agree, and I certainly did not seriously consider rejecting the paper for this. I just was not sure how to address this issue and what to expect from the authors. Reading the paper that half-pass suggested was very helpful. Thanks. Mar 18, 2016 at 17:23
  • 3
    I would like to add that even if the resulting paper is still acceptable, it should be unambiguous in that the hypothesis was formulated after the date was gathered. If it is not clear from the paper, it can unintentionally deceive a reader.
    – Boris Bukh
    Mar 18, 2016 at 18:20
2

Yes! This is statistically a problem, regardless of whether it is accepted in your field or not. If it is accepted in your field, then perhaps reflect on what that means for your field. At a minimum, the authors should accurately describe their two different groups, the two different instruments, and then argue why they can be treated the same instead of omitting this from the reader.

Consider the following thought experiment: Suppose you randomly select two groups of people, and call them the "control" and "experimental" group. Then you apply the same treatment to both groups, for example, giving them both sugar pills. You collect a bunch of health data- blood pressure, weight, height, changes in all of the above, etc.

Due to the nature of probability, you will always measure some effect between two groups. We know, intellectually, that there is no difference between the control and experimental group. But, sometimes the measured effect will be large, which is also due to the nature of probability, even though the two groups have no reason to be different. Sometimes that measured effect will be large enough to pass a statistical significance test, like a p-value test, even though there's no actual reason for the two groups to be different.

Post-hoc analysis is dangerous because you don't actually know why a statistical difference exists. Maybe there is a real correlation, maybe your specific data set just looks that way. For example, maybe your data set shows that the control and experimental groups have a statistically significant difference in their heights. Well, any researcher will throw that out because obviously the treatment cannot possibly effect their height before they came into the study. However, maybe your data set shows that your experimental group had a statistically significant drop in blood pressure over the experiment- Eureka! right? Not really- those two conclusions are equally valid, scientifically. Maybe the blood pressure effect is real, maybe it isn't. You don't know.

If we allow post-hoc analyses, then I go back and change the title of my paper from "I literally treat my control and experimental groups identically." to "Landmark study finds convenient and cheap treatment for high blood pressure." I rack up some great grant funding and continue conducting low-effort, meaningless science with great optics.

This is also called data dredging.

0

The more hypotheses you have, the less power your evidence has to reject wrong hypotheses. This is a very big issue for science in general. It is probably a very minor issue for this paper in particular, but it depends on the paper.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .