18

During a statistical analysis of a study, I found an interesting relationship between two variables that was not part of the original hypothesis. It is, however, clinically interesting and I want to report it in the study.

I've never seen a medical study in which authors admitted to data dredging although I do know it's prevalent. I'm confident enough in my study and methods that I'm willing to admit to it.

What would be the best way to do so in a manuscript? Would I include all the information of the dredged variable such as p-values and confidence intervals, but put a note that this hypothesis is based on the data?

Improve this question
9
  • 3
    Why not consider a second paper?
    – Buffy
    Jan 14, 2020 at 15:29
  • 7
    @Buffy That doesn't really solve the p-hacking problem.
    – Bryan Krause ♦
    Jan 14, 2020 at 17:18
  • 4
    @Buffy Yes but that has nothing to do with whether the analysis is in the current paper or a separate one. Putting it in a separate paper doesn't change that the analysis was unplanned and doesn't provide any control for type I errors. Sure, it's possible that a separate paper is appropriate, but I think it's highly unlikely that this incidental finding would stand on its own, and the OP would like to publish it.
    – Bryan Krause ♦
    Jan 14, 2020 at 17:54
  • 2
    Rather than a second paper, a second study, with adequate power, may solve the problem.
    – user97709
    Jan 15, 2020 at 20:10
  • 2
    Does the result hold when you adjust your p-value for multiple comparisons? If you apply a conservative method like a Bonferroni Correction, that should dispel most accusations of p-hacking.
    – acvill
    Jan 15, 2020 at 21:56

4 Answers 4

Reset to default
47

Yes. I would have a separate section of your paper entitled something like "Further exploratory analysis", report what you did and what you found, and note that until a study has been design to specifically test your hypothesis, it remains a hypothesis, but suggest that it might be an attractive target for further study.

Improve this answer
2
  • 19
    +1. Exploratory data analysis is fine, and hypotheses that are tested in studies don't appear out of the blue and there's nothing wrong with generating hypotheses. It only becomes data dredging when hypotesis generation and testing are mixed inappropriately (claiming confirmation for what is actually HARKing).
    – cbeleites unhappy with SX
    Jan 14, 2020 at 15:17
  • 3
    I don't know about medical papers but in computer science there is often a "Future Work" section which would be very appropriate to mention something like this.
    – Pace
    Jan 15, 2020 at 17:39
16

My best advice is to be very upfront about the fact that

1.) You found some relations in your data that were not apart of your original hypotheses you were interested in testing.

2.) These results relations were still interesting enough to share, although the evidence should be taken with a grain of salt.

Because these relations were found spuriously, the evidence is not as strong as if they were the original hypotheses of interest. When writing this in your results, it's important to reflect this.

In my opinion (as a PhD in statistics, for what that's worth), I'd include unadjusted p-values and confidence intervals, and label them as such; "p-value (without adjusting for data exploration): 0.0013". Thus the reader isn't in the dark about your interesting discovery, but also is not misled about the strength of the evidence.

On a pragmatic note, note that this means this previously unhypothesized finding alone is unlikely to be sufficient for publication, as one could make the argument that the strength of evidence for this finding is not particularly strong. But hitching this result onto the published paper seems quite reasonable if that connection has the potential to be interesting other researchers in the field. One of my professors referred to this type of exploratory data analysis as "hypothesis generating" rather than "hypothesis testing".

Improve this answer
8
  • This is pretty much exactly the advice I give when I get these sorts of questions. +1.
    – Bryan Krause ♦
    Jan 15, 2020 at 5:12
  • doesn't the size of the dataset factor into this? If it's a very large dataset it is less likely to be spurious? E.g. I have access to a million medical records and browse around, and find a very strong link between two variables. That's not the same as a study done on 46 mice.
    – jiggunjer
    Jan 15, 2020 at 13:28
  • 2
    @jiggunjer No, it doesn't. The sample size influences whether a particular correlation is flagged as significant at a particular threshold. But do enough tests and you'll find spurious correlations, at any sample size.
    – Sneftel
    Jan 15, 2020 at 14:20
  • 1
    @Sneftel: actually, it's more complicated than that. What you say is true if you are looking at a dataset in which there aren't really any true correlations. Hopefully, researchers aren't wasting time collecting large datasets where in truth, nothing is related! Now, if you have a mix of true relations and near zero relations, the larger the sample size, the more likely the true relations will rise to the top. So in that context, sample size does matter.
    – Cliff AB
    Jan 15, 2020 at 15:26
  • @CliffAB it is even more complicated as you are also likely to look for more relations in a larger data-set. Both because it is possible, and also because the effort of collecting the dataset might otherwise be seen as wasted.
    – Hans Olsson
    Jan 16, 2020 at 10:59
2

This should be fine, so long as you're doing appropriate multiple hypothesis correction. Note in your manuscript what types of exploratory variables you evaluated for association, and how many of them there were. If your p-value is still significant after multiple hypothesis correction, that means there's still a stronger association than you'd expect by chance alone, which makes it an interesting variable.

If you only report the interesting variable and don't mention the other 1000 variables you tested, you could be rightly accused of p-hacking, which occurs when someone ignores "researcher degrees of freedom" to inflate the significance of their result. There's nothing inherently wrong with testing exploratory variables, you just have to do it in a responsible manner. Pre-selecting variables of study is essentially just a means of using prior knowledge to get around multiple hypothesis correction.

Obligatory xkcd

Improve this answer
13
  • 7
    "so long as you're doing appropriate multiple hypothesis correction": though it may be impossible to do that as OP may not be able to exactly or even roughly quantify the number of hypotheses tested. Still it would be possible to report what exactly was done and what was observed.
    – cbeleites unhappy with SX
    Jan 14, 2020 at 15:12
  • 1
    I don't know why, honestly, but in medicine, there is a ton of retrospective cohort shotgun studies with a lot of variables but rarely do I see the authors adjust their p-values. Often this is followed by "...to protect from type-2 errors".. To be fair often small studies cannot adjust their p-values because it would be impossible to draw any meaningful conclusions, and some specialties are indeed small with a small amount of specific patients.
    – Paze
    Jan 14, 2020 at 15:48
  • 3
    "I can't think of a situation where you wouldn't have any idea how many hypotheses were tested": Andrew Gelman has written extensively about this, calling it the "Garden of Forking Paths". As he argues, we quite often see the data, and automatically start pushing hypotheses up and down, despite not formally testing any of them. Thus, we are informally doing multiple comparisons, with all the same issues, when we first look at our data and then decide what to turn into a t-test, regression, etc.
    – Cliff AB
    Jan 15, 2020 at 4:25
  • 2
    @emory: I'd say we don't, or at least it's not simple. Maybe we look at the data and say that there is clearly a quadratic relationship so we test that, or notice an obvious trend that has an outlier so we inspect that outlier and remove it, etc.
    – Cliff AB
    Jan 15, 2020 at 15:15
  • 3
    @emory: operationally, that approximation accurate enough :). Sorry to keep driving on this, but Gelman's point on the topic is that we are not aware of the number of forks. When we look at the data, we are not aware how many possible paths ("oh, this needs a log-transformation", "clearly these terms are interacting!", etc) we could have walked down had the data looked different.
    – Cliff AB
    Jan 15, 2020 at 17:21
0

Maybe talk to your supervisor about it. I had something similar in my data set and it turned out that it was something that was not significant, as in it was the exact relationship a more experienced scientist would have expected to see, so I would be stating the obvious.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .