31

I tested a hypothesis using some observational data, but did not find support for the hypothesis (the difference was not significant at the p < 0.05 level). I subsequently realised that the inclusion of a certain subset of individuals was dubious, since there was some doubt over their measurements. I filtered out those individuals (on objective criteria), and lo and behold, I now find evidence to support my original hypothesis.

I am acutely aware that, had the original hypothesis test returned a significant result, I probably would have found justification to support the inclusion of those doubtful measurements (or perhaps never even stopped to think about removing them). Although this wasn't my deliberate intention, what I have done seems a lot like I have employed my “researcher degrees of freedom” to find a version of the analysis that supports my original hypothesis.

I realise that what I should have done was to plan out my analysis more carefully in advance, and decide whether or not to include the doubtful measurements before carrying out any analysis. But I can’t change what has already happened, so my question is, what do I do with my data/analysis now? I can see a number of options that vary in their sensibleness, but none of which seem ideal:

  1. Continue to use my updated analysis and present a clear argument for why those individuals should be excluded (i.e. ignore the RDF issue).

  2. Continue to use my updated analysis but, in any write-up/publication be fully open about the less-than-ideal path that I took to reach it.

  3. Conduct some kind of multiple-comparisons adjustment to take my multiple different analyses into account (I don’t know if this is even a valid approach in this context).

  4. Sigh and throw the analysis in the bin.

How (if at all) can I make good use of my analysis while still adhering to good research practice?

My field is Ecology, in case that makes any difference.

Edit, in response to close vote: I did consider submitting the question to Stats.SE instead, but I felt that this is more an issue of research philosophy than a detailed statistical question. The answers that I am interested in (and indeed have already received) are connected with a general research strategy, and how to present results, rather than details regarding particular statistical methods, and so I felt that it was appropriate for this site. Having received very useful answers and comments already, it won't make too much difference to me personally if it ends up closed (and I can understand the argument for doing so), but I think the material could be useful to others in a similar situation to my own.

  • 2
    Can you obtain more data? testing your refined analysis on new, independent data is the ideal solution, if it's applicable. – Trisoloriansunscreen May 5 '16 at 13:47
  • 17
    Keep in mind that the .05 cut-off is totally arbitrary. A p-value of 0.04 is showing basically the same thing as a p-value of 0.06, yet one is trumpeted as a "significant" finding and the other is rejected as a disappointing negative result--this makes no sense. Try to get away from a focus on significance in the way you report your findings. – user24098 May 5 '16 at 14:01
  • 4
    You didn't get a negative result. You concluded that you shouldn't reject the null hypothesis. And if you're using a hypothesis test (unless it's a test for trend, which is a rather different thing) there's no 'trend' - you either reject the null hypothesis or you don't. If you're testing with 5% significance and p = 0.06, the null hypothesis stands. There's no 'trend towards significance' - that's a meaningless concept. If you don't like that, find a different analytical model. – rhialto May 5 '16 at 17:01
  • 6
    I'm voting to close this question as off-topic because it is a matter for the statistics SE – Scott Seidman May 5 '16 at 18:10
  • 3
    I don't agree that this is off topic here. I will vote to reopen, if it gets a few more reopen votes. – ff524 May 6 '16 at 9:15
35

Clearly report what you did and both results. Readers will make up their own minds about the strength of the findings.

The situation that you have encountered is very common. While testing a prior hypothesis with predefined methods is the right thing to do, in practice it is unavoidable that you will sometimes find problems that you need to correct after running an initial analysis.

Also, if your analysis produces an unexpected result, it is quite logical that this will lead to extra checking of what you have done, and further analyses to confirm it.

There's nothing wrong with doing extra things to refine the analysis. The problem would be refining the analysis and then failing to report on earlier versions.

A multiple testing correction would not typically be used for only two analyses. Furthermore, it is a judgement call whether this counts as two tests or one. If the second analysis was really the "correction" of something flawed, that is not quite the same as repeating two different but valid approaches. And there is not a clear line as to when that becomes true.

In summary, there is no problem with what you have done, as long as you report it, but your concern with doing this the right way is a very good thing--please keep it up! Inadequate reporting of this sort of thing is quite common, and it undermines the quality of scientific evidence.

7

@dan1111's answer is excellent if you do not have the resources to repeat your experiment, but under ideal circumstances, no matter how your model was created, an independent replication of the experiment would be the best possible evidence of your model's strength.

Much like in experiments involving fitting models to data (such as protein crystallography or data mining), it is perfectly acceptable to refine/train a model after obtaining your initial data, if the model is then fitted with independent results from a different dataset.

For example, you could repeat the data collection (and apply your new filter), and then run your model on the new dataset. If the new dataset confirms the previous one, your model would then be pretty solid.

In any case, no matter which path you take, you should present all of your results in your final publication. If you do in fact have the resources to repeat the experiment, the writeup could go as follows:

I collected dataset A, and then performed analysis using filter 1. However, the results were not statistically significant.

Upon realising that there were a number of flaws in filter 1, I then modified the filter, removing (anomalous values) and changing the filter to filter 2. Upon repeating the analysis, the results were statistically significant.

To reduce the likelihood of "researcher degrees of freedom" affecting the model, I then collected an independent dataset B. Upon repeating the analysis using filter 2, I found that the results replicated the findings of Filter 2 on Dataset A.

1

dan1111's answer is quite good, and March Ho's answer is also very valuable.

Even though this is a 9-month-old thread, it remains a timeless topic, and so I would like to submit a further thought on the topic for you and/or any other researchers who stumble upon this question.

You asked whether researcher degrees of freedom invalidate your analysis. "Invalidate" might be a bit strong, but it certainly casts serious doubt on the effect you're investigating. Further complicating matters is that, in certain fields (and particularly in certain segments of certain fields), such practices are basically standard operating procedure.

See the story on p. 1 at Nosek, Spies, & Motyl (2012) doi:10.1177/1745691612459058 for a similar kind of scenario. The answer, as I see it, is that you should really treat your results as exploratory, and maintain a healthy skepticism about them. If you want to increase your certainty (and it sounds like you do), then the best answer is to run replications.

Note the plural here. March Ho advocates an independent replication--that is, a single replication. While this can be highly useful, as in the example in Nosek, Spies, & Motyl (2012; linked above), it can lead to further ambiguity if the replication yields a similar, but weaker, effect. Therefore, a single replication may backfire by increasing your uncertainty rather than your certainty! The ideal would be to test the effect multiple times, and look at the results in aggregate. Then, you will be much better able to 1) determine the direction of the effect at the population level, and 2) estimate the magnitude of the effect at the population level.

This approach would take a long time, possibly a lot of money (depending on the nature of your research), and it wades into the murky debate over whether a direct replication or a conceptual replication is more valuable. Despite these limitations, however, the advantage of this method is that you'll end up with a very high degree of certainty about the nature of the effect that you're studying.

If your goal is to increase certainty in your findings, I don't see any viable alternative to conducting multiple replications (both direct and conceptual). If your goal is to generate new findings and you're not terribly concerned about the generalizibility of the results, however, then the exploratory method that you described isn't much of a problem.

At least some academic journals seem to prioritize novelty over certainty. But people who care about the integrity of science seem to prioritize certainty over novelty. I think each approach can be valuable, but not at the exclusion of the other.

For maximum transparency while also maintaining publishability, it would probably be best to follow March Ho's above answer, which provides a great template to solve the issue if further data collection isn't possible.

This is a particularly thorny question! Research methodology and statistical methodology are inextricably linked. I have written about best statistical practice, on a note that is related to this issue; a preprint of the under-review manuscript is available at http://osf.io/preprints/psyarxiv/hp53k/

0

What does your pre-registered protocol and statistical analysis plan say? You should do that.

If you don't have one, then make sure you do next time.

For the write up here, I think you'll have to present both analyses and explain why you did both.

You should also get away from the belief you seem to have that only results exhibiting a statistically significant result are publishable. That's not the case (see eg https://mobile.twitter.com/trished/status/723202381622730752 from a journal editor). Reputable journals look at the question addressed and the methods used. If you've asked a good question and done the experiment well, then demonstrating the absence of effect is an exceeding positive and publishable finding.

In medicine, that sort of finding would save the country lots of money and spare patient from undergoing needless treatment. I've no idea what would happen in ecology, but publishing it would at least stop your colleagues going down the same fruitless route.

  • As stated in my question, I understand that it would have been helpful to have a clearer protocol decided in advance, but that does not help me in the present situation. Also, nowhere have I stated or implied that I only care about results that are statistically significant. Sorry to be defensive, but it's only really your third line that addresses my situation. – user2390246 May 5 '16 at 21:07
  • 2
    Negative findings should be just as publishable, but in practice this is not the case. It will be quite hard to get them into most good journals. – user24098 May 5 '16 at 22:42
  • 1
    Once again, referring to a finding of no effect as a negative result is far from helpful - and probably contributes any difficulty in publishing. A definitive finding of no effect is a very positive result. – rhialto May 5 '16 at 23:20
  • I have edited to clarity this point - thanks. My original wording was because I was trying to keep from getting into statistical detail, given this is Academia.SE not Stats.SE, but I can accept that this led to some unhelpful implications. – user2390246 May 6 '16 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.