6

According to this paper by the previous editor of Journal of Finance, Campbell R. Harvey, authors of papers should "convince the reader that there has been minimal data mining". Why is it so?

  • 4
    They have confused data mining with hypothesis mining. – Anonymous Physicist Feb 5 at 22:33
  • 3
    @AnonymousPhysicist, probably an answer. Different fields use terminology differently. – Buffy Feb 5 at 22:58
  • What do you mean by data mining? If it is sifting through all the data to find what agrees with your model, you are doing it wrong (as one example of usage). – Jon Custer Feb 5 at 23:08
  • 5
    It should be pointed out that "data mining" is sometimes used as a pejorative to describe the process of testing large numbers of hypotheses in hopes of finding one that is statistically significant so that you can publish a significant result. This is a completely different sense of the phrase from its other use to describe the process of building a predictive model that is tested for its predictive ability on out sample data and then used to make predictions on new data. That second sense is what data scientists do. The first sense is what bad scientists do. – Brian Borchers Feb 6 at 3:18
  • Data minimization is an important aspect of privacy engineering. When the research relies on data that might have privacy-related implications, it's important to not collect more data than required for the purpose of the research. – lighthouse keeper Feb 9 at 7:29
13

Assuming by "data mining" what they really mean is "continually testing hypotheses until you find something significant"...

See:

https://stats.stackexchange.com/questions/310119/why-does-collecting-data-until-finding-a-significant-result-increase-type-i-erro

or a comic depicting the same issue:

https://xkcd.com/882/

Basically, when we perform a hypothesis test (using a frequentist approach), we are asking whether it is likely that the data we observe (for example, a difference between two groups) occurred simply by chance.

However, if you look at multiple possible questions, it becomes increasingly likely that you find a result that, due to the flawed approach, appears unlikely to have occurred by chance. If you use a significance threshold of 0.05, you in fact expect that 1/20 times (in the large number limit) you look at completely random data you will conclude a significant difference.

You can still do this systematically and correct for this exploration without increasing your false positive rate. The problem is when you set these explorations aside and pretend that only the "interesting" ones (based on a flawed measure of statistical significance) occurred, you are likely to report a result that occurred by chance and is not replicable.


This problem is even worse if you use your results to inform further analyses in an unprincipled way. For example, you compare two groups: jelly bean eaters and non-eaters, and find no significant difference in cancer risk, but you look and see that the difference is smaller in a subgroup. You then exclude that subgroup (say, people over 55 years old), repeat your analysis, claim "statistical significance", and write a paper titled "Jellybeans cause cancer in people under age 55".

If you didn't set out to test this hypothesis because you had no a priori reason to expect age differences in jelly bean danger, it is very likely your conclusion is wrong: you have "mined" out a result from the data that didn't really exist in the first place.

| improve this answer | |
  • 1
    Nitpick: It should be "it becomes increasingly likely that you find a result that, due to the flawed approach, appears unlikely to have occurred by chance." – Roland Feb 6 at 13:02
  • @Roland Thanks; fixed. – Bryan Krause Feb 6 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.