How to argue against questionable research practices such as P-hacking and Harking?

Question

I have come into conflict with co-authors when being asked to do things that I consider to be questionable.

1. Once I was told to try every possible specification of a dependent variable (count, proportion, binary indicator, you name it) in a regression until I find a significant relationship. That is it, no justification for choosing one specification over another besides finding significance. The famous fishing expedition for starfish (also known as P-Hacking).

2. In another occasion I was asked to re-write a theory section of a paper to reflect an incidental finding from our analysis, so that it shows up as if we were asking a question about the incidental finding and had come up with the supported hypothesis a priori. The famous hypothesising after results are known (Harking).

In both cases I refused to comply and explained my reasoning, what led to conflicts with the other party. I tried my best to not sound accusatory (not to give the impression that I doubt the ethics of the other party), but it nonetheless led to attrition and a worsening of the working relationship. In the long argument that followed, I was told that 'social science is not done as the natural sciences,' and that I was 'too inflexible,' 'too positivist,' and that everybody does these things that I was being asked to do. The argument culminated with me being asked to 'stop obstructing the progress of the paper,' what made me feel very frustrated.

Since then I have seen several cases of what I suspect to be this type of research practice. For example, PhD students coming to me to ask about what they should change in their models so that their results come out significant, and people working at the same computer lab as me asking me for the same type of help.

I do consider these things to be seriously questionable from an ethical point of view, and would like to be able to argue against them effectively. However, the other parties are usually experienced researchers or students under the supervision of an experienced researcher. As a young researcher, I feel that I'm at a disadvantage when arguing against. It is often the case that I'm arguing against the instructions of someone who has more experience, publications, and, supposedly, knowledge than I do.

Is this one of those cases where we can't do much but try to be the 'change that we want to bring about,' shud it, and just make sure that we are doing the right things ourselves? Should we speak up more often? If so, any good strategies to be more effective and convincing?

p.s. The tag is social sciences because of my field, but I reckon that this happens in other areas as well, and I welcome input from other fields.

---

EDIT 1: In example 2), at no moment anyone suggested that we would confirm the new hypothesis in a new set of data. The intention was to pretend that we got it right from the onset, which is why I objected.

EDIT 2: Just to make clear. I am aware of the right way of doing these things (i.e. cross validation, confirmatory analysis in a new dataset, penalising for multiple statistical tests, etc.). This is a question about how to argue that p-hacking and harking are not the way to go.

EDIT 3: I was unaware of the strong connotations of the word misconduct. I have edited and replaced it with 'questionable research practices'

Answer 1

Kenji, For the last few years, I have given a continuing education course called Common Mistakes in Using Statistics: Spotting Them and Avoiding Them. I hope that some of the approaches I have taken might be helpful to you in convincing your colleagues that changes are needed.

First, I don't start out saying that things are unethical (although I might get to that eventually). I talk instead about mistakes, misunderstandings, and confusions. I also at some point introduce the idea that "That's the way we've always done things" doesn't make that way correct.

I also use the metaphor of "the game of telephone" that many people have played as a child: people sit in a circle; one person whispers something into the ear of the person next to them; that person whispers what he/she hears to the next person, and so on around the circle. The last person says what they hear out loud, and the first person reveals the original phrase. Usually the two are so different that it's funny. Applying the metaphor to statistics teaching: someone genuinely is trying to understand the complex ideas of frequentist statistics; they finally believe they get it, and pass their perceived (but somewhat flawed) understanding on to others; some of the recipients (with good intentions) make more oversimplifications or misinterpretations and pass them on to more people -- and so on down the line. Eventually a seriously flawed version appears in textbooks and becomes standard practice.

The notes for my continuing ed course are freely available at http://www.ma.utexas.edu/users/mks/CommonMistakes2015/commonmistakeshome2015.html. Feel free to use them in any way -- e.g., having an informal discussion seminar using them (or some of them) as background reading might help communicate the ideas. You will note that the first "Common mistake" discussed is "Expecting too much uncertainty." Indeed that is a fundamental mistake that underlies a lot of what has gone wrong in using statistics. The recommendations given there are a good starting point for helping colleagues begin to see the point of all the other mistakes.

The course website also has links to some online demos that are helpful to some in understanding problems that are often glossed over.

I've also done some blogging on the general theme at http://www.ma.utexas.edu/blogs/mks/. Some of the June 2014 entries are especially relevant.

I hope these suggestions and resources are helpful. Feel free to contact me if you have any questions.

Answer 2

This sort of thing happens in both the social sciences AND physical sciences. For instance, often a scientist will collect data to test a theory but will also collect lots of extraneous data. Analyses on these extraneous data often should be considered exploratory and labeled as such (because significant results could be due to the multiple tests) [As another example, you don't want to know how often chemists repeat an experiment until they get a good yield, then stop and report that yield without mentioning that that was the best in 20 experiments!]

The fastest solution is to agree to do the multiple analyses, but then tell what you did in the methodology section. If you say that you analyzed it several ways and one way showed significance, readers can decide whether or not to believe the result. Just tell your co-authors that not mentioning that you did multiple analyses is leaving the research improperly described.

However, you can (occasionally) save the day. If, for instance, you did 10 different analyses and picked the best one, you'll be ok if the result would hold under a Bonferroni correction (i.e. instead of requiring significance at the 0.05 level, you require significance at the 0.05/#tests level). So it the final test shows a p-value such as 0.000001, you probably are on safe grounds.

Another approach is to a priori decide that some tests are obvious (confirmatory) and some are just searching around the data (exploratory). Then you can demonstrate the confirmatory results, while labeling anything interesting among the 'exploratory' results as 'needs further research'. That is, you can mix well-founded tests with 'data dredging' as long as you acknowledge the difference between the two sets of tests.

But if it isn't possible to rescue the result, I'd go with insisting that they describe what they did, with the comment that if they are embarrassed to describe it, they shouldn't have done it. :)

You might also add that it is often obvious (at least to statisticians) that a researcher has pulled this trick. When we see a test in isolation that would not occur to us to be the obvious approach, or a hypothesis that we'd not choose a priori, it looks suspicious. For instance, I recently read a paper that claimed that a certain group of people tend to commit suicide more often if they were BORN in the Spring. It was clear that JUST testing the effect of birth in Springtime was not something that would occur to anyone, without testing the effect of birth in other seasons. So they probably had a spurious result due to multiple comparisons.

Answer 3

This is an excellent question. I do think you (and others in similar situations) should speak up, but I realize this is very difficult to do. Two things I'd suggest:

1. Try to figure out if the people you're dealing with understand that the methods they're proposing (p-hacking, etc.) are dodgy or not -- i.e. whether it's an issue of ethics or ignorance. This is harder than it may seem, since I think many people genuinely don't understand how easy it is to find patterns in noise, and how "researcher degrees of freedom" make spurious patterns easy to generate. Asking people, non-confrontationally, to explain how doing tests on "every possible specification of a dependent variable" and selecting those with "p<0.05" corresponds to <5% of "random" datasets having a feature of interest would make this clearer, and would perhaps give you insight on the question of ethics or ignorance. I'd bet that a good fraction of people aren't deliberately unethical, but their cloudy grasp of quantitative data obscures ethical thinking.

2. Something I've found helpful in related contexts is to generate simulated data and actually show the principle that you're arguing. For example, generate datasets of featureless noise and show that with enough variables to compare between, one can always find a "significant" relationship. (Obviously, without correcting for multiple comparisons.) It may seem strange, but seeing this in simulated data seems to help.

Good luck!

Answer 4

Your instinctive concern about creating hypotheses out of data and pretending they were there from the outset is on the right track:

In statistics, the so called chi-square test can be used to compare data with models which have been fitted out of the data themselves. However, for this, the chi-square test must be adapted to essentially "penalise" one's extraction of the parameters when testing how significant the match is.

This is not easily generalised to other setups, so in general learning theory and practice, one splits the data into multiple groups. For example, where one part is used to optimise the parameters, one, at first unseen, part is used to optimise the generalisation, and the last, unseen, part never feeds into the model construction and is used to test how well the first two stages worked. This is called "cross-validation".

Perhaps you can suggest (or simply introduce) to your group such a methodology, by splitting the data randomly into different components; out of one you construct the model, which then is tested with the unseen data. Details of how to do the split would depend on your domain. This way, you have the confidence that the model is predictive. For this to be sound, you need to make sure that it is not using the complete dataset in any form (not even through one smart colleague that remembered that the data are parabolic on the whole). Best is to not ever look at the unseen data until the model is complete.

As for post-hypothesising, I found this often not even to be necessary. You might start with a hypothesis, then discover it is not valid, but then find another, interesting phenomenon instead. This is called "discovery" and the coolest papers result from that. If the top journals of your field do not accept such a style, because they want the standard "hypothesis-experiment-validation" cycle, then the problem lies deeper in your community than with your colleagues.

In short: fitting models out of your data and comparing match is ok if you have a way of penalising that extraction (as in the chi-square). Failing that, you can do "cross-validation" for sound results. Finally, instead of post-hypothesising, my suggestion is to hypothesise, say, invalidate the hypothesis and demonstrate the emergence of a different hypothesis.

Answer 5

Describe exactly what you have done in the paper. As long as you are honest, the paper will be judged by the reviewers, editors and readers.

Even people doing p-value hacking will have a hard time removing an honest description from the paper. If they tell you to remove it, ask them why and you will have the upper hand in the resulting discussion.

Answer 6

Lots of good answers already. However, in academia, it's always better if you can back up your position with a nice published reference. Happily, the question of p-hacking and replicability is being raised and addressed more and more often in different disciplines.

I'll set this up as a CW post to collect pointers to relevant publications we can use in discussions with coauthors that don't see the problem with questionable statistical practices. Everybody, please feel free to edit with your discipline's relevant articles or conference papers.

Psychology

• Here is an editorial by the Editor-in-Chief of Psychological Science, which is pretty much the mother of all psychology journals (Open Access. I also recommend papers cited by Lindsay.):

  Lindsay, D. Stephen (2015). Replication in Psychological Science. Psychological Science, 26, 1827-1832. DOI:10.1177/0956797615616374.

• Here is a study in Science that shows that we indeed have a "replicability crisis" in psychology - a large collaboration set out to replicate 100 effects reported in well-regarded journals, and only 36% did replicate:

  Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349, 6251. DOI:10.1126/science.aac4716

Answer 7

One option is to make 'constructive' points. If your co-authors are (as many are) used to different degrees of p-hacking, they will probably not be too happy to hear that their results are unpublishable as they stand.

If you were able to offer a solution to publish the results while also avoiding these bad practises, then few would object. The best way will probably to try out doing bayesian analyses. Here, (in some cases) non-significant results will also be interpretable and thus publishable.

Answer 8

I hope you don't mind, but I want to take this chance to give you a different set of advice from what you are asking for - to advise you not to take this approach to tackling this issue at this point in time.

I am making the assumption that you want (i) some level of academic success - enough to support yourself, and (ii) to improve the quality of research, and the social benefits that result. Assuming I am correct, I don't think you should pursue this argument (at this stage).

I don't think you should pursue this argument (at this stage) if you value your academic career as it will cause you to burn important bridges and close doors. For instance, if your supervisor p-hacks and you expose and destroy him for it, then you will have lost your main support and dramatically reduced your likelihood of being able to secure a career in this area.

As related to this, (at this stage) I don't really think that you are optimally placed to challenge the negative influence of p-hacking. Here are a few reasons why I think this. First, as related to what I said above, if you are in the infancy of a career within a specific social system then you cannot easily impact the behaviour of those who are already established in that career, and who neither know nor respect you. Second, you cannot fully understand why the system operates as it does, nor the levers that need to be pulled to change that operation, until you are more familiar with it. You might be able to make a micro level difference (e.g., you expose some people you work with), but I don't see that as likely to be very effective, as it will cripple you to do so.

To sum up my thoughts with an anecdote: imagine that you grow up in a city where all the teachers are corrupt and incompetent. Do you think it would be best to protest against them when are in the school? Probably not, as you would achieve very little, and the teachers would probably use their power to prevent you from graduating and essentially ruin your life. Alternately, would it be better to tolerate the teachers flaws until you are out of the system (or higher up) and in a position to actually change things? I would think so as in that case you might end up in a position of authority, and have the resources available to do something to change the teaching system.

Of course, all of this is just my opinion and I could see many ways in which you could argue against it :)

28/12/2015: Adding some more content to explain and address comments.

I would like to put more emphasis on my main point; to appeal to pragmatism and wait for a better time to act. Personally, I think that there is a time and a place for activism, that sometimes it is best to keep your mouth shut and wait until you have a better chance to do something rather than to speak up and get shot for nothing. Thus, in any case where activism is an option, the decision whether to engage in it should be contingent on various consideration, such as the severity of the undesired outcome, the risk to the individual in preventing it, their ability to prevent it, and their moral framework (e.g., deontological or utilitarian). As the saying goes, you need to pick your battles; every battle will take its toll and some tolls might not be worth paying for what they get you.

Personally, I feel that if you are going to publish something that might wipe out humanity or end up with someone getting killed then by all means you should make a personal sacrifice to prevent it (if you can do something). On the other hand, if the current 'negative' outcome that you foresee is unethically (by some/most authors current norms) changing the focus of a paper (that 5 people will actually read) to look at one significant relationship (e.g., age and correlation to frequency of cycling) rather than another previously planned relationship (e.g., gender and correlation to frequency of cycling) that turned out to not be significant, and the new outcome is either (i) you will be sacked and the paper published without you, or (ii) nothing will be published and no-one will ever benefit from knowing about the significant relationship that you found, then I am more convinced that engaging in activism is not the way to go (at this stage anyway).

And yes, I accept that my arguments here are flawed simplifications of what is a very complex reality, but I hope you can understand my general point and give some thought to it.