3

The question sounds bad but here is the context:

My area of focus is psychology where preregistering studies (e.g., methodology, hypotheses, analytical plans) is becoming more and more popular to avoid issues with p-hacking or HARKing (hypothesizing after results are known). Preregistration has garnered the endorsement of many prominent psychologists and it's now being encouraged by many top tier journals.

However, I've seen less than ethical behaviors among some behavioral scientists in which they attempt to still p-hack or HARK while getting the preregistration badge.

For instance, suppose that John decides to run an exploratory survey study on various topics with no real a-priori hypothesis. In this large dataset, John finds an interesting effect and thinks that this finding is publishable. He develops a post-hoc conceptual reason for why the effect may exist. John wants to publish in the journal Psychological Science. However, this journal encourages preregistering of studies. In order to get the preregistration badge, John preregisters a new study with a hypothesis on the effect that he found, making it seem like he had a theoretical and conceptual reason to develop that hypothesis (rather than it being from an exploratory analysis; basically the act of HARKing). Because he never published anything from the exploratory dataset, no one else knows about John's HARKing.

Sidenote: this is a relatively extreme example--many journals are okay with an exploratory, unregistered study if it's also followed up with preregistered, confirmatory studies.

In this scenario, one could easily bypass the preregistration honor code and HARK without much chance of ever being caught. In other words, while preregistering has good intentions, I'm under the impression that it also can result in people becoming more creative in their endeavor to p-hack and HARK.

I'm hoping to create workshops in my department regarding how preregistration is a good approach to minimizing issues in modern psychological research, but can also present a new flurry of issues. To do this, I'm trying to get a good sense of how researchers have found loopholes for preregistration. What kind of methods have you observed / engaged in / heard of / or can reasonable figure would bypass the preregistration process?

DISCLAIMER: I study ethics and decision-making so I mainly aim for behavioral economics and business journals where preregistration is not as strongly endorsed as in social psychology. If I misunderstood some part of the preregistration process or norm, do feel free to correct me.

4
  • 5
    Frankly, yes, this is a limitation of many kinds of pre-registration. The only way to avoid it is to use data you can show did not exist at the time of pre-registration. Dec 7, 2019 at 23:27
  • 4
    From a cryptographic/security viewpoint, it would not be an easy thing to prove that something did not exist earlier... It already requires belief in peoples' honorable behavior to believe their reports on their own honorable behavior in this regard. Dec 7, 2019 at 23:56
  • @paulgarrett For some kinds of data, that's fair. On the other hand, the provenance of some kinds of data (for example, I work with human neuroimaging) would be fairly trivial to prove post hoc it was acquired post-preregistration if someone really held your nose to the grindstone, considering funding submission, human subjects approval, and facility logs. Dec 8, 2019 at 4:01
  • But yes, preregistration is ultimately based on trust, as much as the value we place in non-preregistered publications. Dec 8, 2019 at 4:03

1 Answer 1

1

The value of preregistration for a retrospective study is pretty marginal. Preregistration works best for prospective studies where the primary analyses and hypotheses to be tested are finalized before data collection begins. It is worth noting that preregistration does not prevent data fabrication or many other types of scientific misconduct. The main benefit of preregistration, beyond making the process more open, is by preventing p-hacking and HARKing. This only works for new data, or at least for new aggregations of existing data.

The worse abuse of preregistration I have seen is a study where the hypothesis and planned analyses where changed 6 months after data collection was completed. The paper then talked about the planned comparisons from the updated version of the preregistered study.

1
  • Correct me if I'm wrong, but doesn't preregistration allow you to make revisions if the original hypothesis/analysis does not work out as planned as long as you are transparent of this revision in the publication?
    – ssjjaca
    Dec 10, 2019 at 22:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .