Are P-hacking and hypothesising after results are known considered misconduct in academia?

Question

This question is inspired by a comment to another question where I asked for help on how to argue against P-hacking and hypothesising after results are known (Harking). Someone questioned the classification of these two behaviours as misconduct, and my general experience (around my close academic circle) is that many see these two activities as part of the way we do science.

Here is what I refer to by P-hacking and Harking.

P-hacking is when someone collects more data, changes the specification of a statistical model, change the analysis sample, or does other changes to the study until the results become statistically significant. Many of these things can be done with a justification, but the p-hacker (p-fisher) does them solely with the intent of obtaining a significant result. In doing so, he or she risks capitalising on statistical error (type 1 error) and publishing results that are basically a false positive.

Harking is when someone generates a scientific hypothesis about the data after seeing the data. It would be innocuous if the researcher acknowledged the exploratory nature of the study and sought to confirm the findings in another set of data (or if he or she used cross validation techniques). It becomes a problem when researchers pretend that they had the hypothesis a priori and that the study was done to confirm it, hiding the exploratory nature of the study and conferring more strength to the results than they actually have.

I am not asking for opinions on whether these things should or should not be considered misconduct. Rather, I would like to know the overall position of scientists in fields where statistics are used. I know of no survey on how scientists view these two behaviours, but I welcome answers that include such data.

Answer 1

The Declaration of Helsinki was updated in 2013 to "mandate" that research involving human subjects must be pre-registered. While not perfect, pre-registration prevents many of these statistical manipulations. The idea of pre-registration is that publicly stating your hypotheses and the details of how they will be tested in advance reduces questionable statistical practices. For example, changing the number of subjects, the inclusion/exclusion criteria, or the statistical model are not allowed.

From my understanding, failure to comply with the Declaration of Helsinki would be considered unethical in Medicine, while in other fields the pre-registration aspect is being actively ignored. For example, articles are now being published with disclaimers like "this research was conducted in accordance with the 2013 Declaration of Helsinki except the study was not pre-registered".

Answer 2

In the US, for research funded by the NIH, "Research Misconduct" is a finding made by the NIH Office of Research Integrity. Other federal agencies have offices with similar responsibilities for research misconduct.

The ORI web site has a "RCR Casebook" of fictionalized example cases used in training researchers about responsible conduct of research and research misconduct. It also has case summaries for every case where misconduct was determined and administrative penalties are currently in force (that is, cases where someone has been barred from getting NIH funding for some period of time.) In my reading through the training materials and case summaries, I haven't seen any cases where p-hacking was found to constitute misconduct. The cases are much more about outright fabrication of data or suppression of inconvenient data (e.g. by throwing out "outliers") to achieve a desired result. It appears that from the ORI point of view p-hacking is not (yet) considered research misconduct.

More on what it takes to reach the level of "misconduct." The NIH recognizes three kinds of research misconduct:

Fabrication: Making up data or results and recording or reporting them.

Falsification: Manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.

Plagiarism: The appropriation of another person's ideas, processes, results, or words without giving appropriate credit.

p-hacking wouldn't fit under "fabrication" or "plagiarism." It might count as "changing or omitting results such that the research is not accurately represented in the research record." However, the ORI also requires that:

There be a significant departure from accepted practices of the relevant research community; The misconduct be committed intentionally, knowingly, or recklessly; and The allegation be proven by a preponderance of the evidence.

That's a pretty high standard. I think it would be hard to make the case that p-hacking is a significant departure from accepted practice and furthermore a researcher could claim that they didn't intentionally do the p-hacking.

Answer 3

These types of data-straining behaviors are most certainly scientific sins, in the sense of being stains on one's conscience and reputation. My favorite discussion of such sins is the "Nine Circles of Scientific Hell."

Building a formal misconduct case around such data-straining would likely be very difficult, however, since they may quite easily and naturally arise from human propensities to fool ourselves. Many people who engage in de facto p-hacking are not aware of it, particularly when there are large volumes of data and powerful analytic tools in play. A beautiful illustration of both the problem and an appropriate scientific response is the wonderful study that used popular fMRI methods to localize cognitive functions in the brain of a dead salmon.

Answer 4

There's a nice paper showing the prevalence of questionable research practices. For example, more than 65% of researchers who participated admitted to not reporting all dependent measures: https://www.cmu.edu/dietrich/sds/docs/loewenstein/MeasPrevalQuestTruthTelling.pdf

Unfortunately, they didn't ask about p-hacking with things like multiple analyses. I don't think running multiple specifications is a problem, however, and I think it's quite natural to do so.

To give a personal example: I once recoded a three-level variable in an experimental study into a binary response, as I was primarily interested in a treatment effect on one of the responses. This was transparently disclosed in the paper. However, a reviewer asked that I instead do the analysis with multinomial logit as that was more appropriate, even if it comes at a cost of making the coefficients more difficult to interpret (given the importance of communicating results, I think this is something that needs to be given some weight as well: the target audience would not be familiar with multinomial logit). So I redid the analysis as requested and dropped the logit regressions from the paper.

It turned out that the results were now stronger than before, so this adjustment worked in my favor. Suppose I had realized on my own that multinomial logit was more appropriate prior to submission. Would that have been p-hacking?

Consider another (fictitious) example. Suppose I have a regression with repeated observations from each participant and my treatment effect is significant if I use random effects at the individual level, but does not reach significance if I use fixed effects (or vice versa). In many analyses, either could easily be defended -- which one is "correct?" It can't just be the one I happened to choose first. It's a bit like using AIC or BIC for model selection: sometimes, one suggests the model provides a better complexity-adjusted fit, while the other suggests it doesn't. I don't think one is inherently better than the other.

The solution is to ask for more robustness checks in the analysis. Instead of showing only a model with an interaction, for example, the model without an interaction could be shown next to it. (This is actually quite common in empirical economics, where the goal is to show that the result holds under many possible specifications -- and not necessarily that one has found the one true specification.)

Pre-registering research makes sense in medical trials that are pretty straight forward: Group A gets Treatment X, Group B gets Treatment Y, and we compare outcomes on some dimension. If we compared the groups on 30 possible outcome measures, we'd likely see a difference in one of them. That obviously cannot be sufficient to establish efficacy.

But social science research is much more iterative and just doesn't work in the same way. Moreover, most recent papers report multiple studies that back up a particular claim. While the effect may still not be real, there are so many other things that are likely to be more problematic than the model specification. For example, there has recently been some work on incentives for creativity -- e.g. do people become more creative when you pay them more. Imagine the hundreds of different ways you could define and measure "creativity" and the hundreds of settings you could test this in (individuals? groups?). All of those are judgment calls and we won't know what generalizes until there are a dozen of these experiments (ideally by many different researchers).

Harking, it seems to me, is at least in part the result of how journal articles are written. They are not meant to be chronological accounts of one's thoughts, exploratory analyses, and eventual conclusions. They have to succinctly place work in a broader literature and convey the contributions of the present work. It may well be that prior to running the experiment, two theories would have been equally plausible -- but after running the study, only one of those is reaffirmed. If so, it seems like a natural stylistic choice to set up the study using the theories that are consistent with the results, then note in the discussion that the findings go against other possible explanations. (Including alternatives that the researcher may not have thought of, but that a reviewer connected the study to.)

Answer 5

A counter example is Grounded Theory.

It is a methodology used to build a theory out of data where you constantly try to match a hypothesis to your data. So your question about hypothesizing after the fact is definitely not unethical in this case!

It is worth noting that this is probably one of the few exceptions to the problems that the OP and other answers have been aimed at.