I want to ask whether changing p-values adjustment method after I receive major revision is acceptable. I have performed some statistical test for like thousands of elements in the manuscript, and initially I used benjamini-hochberg adjustment and wrote the manuscript.
However, upon revising, I realized that there are many spurious results that very subtle change are considered significant. If I changed it to bonferroni correction, the result seems 'reasonable' i.e. spurious results are not considered significant.
Is the change like this accepted for major revision? In my opinion it is acceptable as we are changing to more stringent criteria (not bonferroni to BH) and the conclusion of manuscript does not change, however, I want to hear your opinion that how you feel if you are in this kind of situation as a reviewer.

Thanks in advance

  • 2
    Should move this to cross validated. I would wonder why you are switching to a more conservative.approach, if the only reason was that you originally got results you didn't like (i.e. you thought they weren't "reasonable") I wouldn't consider that a good approach. What did the actual reviewers say? – RAND Dec 24 '19 at 2:46
  • 1
    @RAND Thank you for the reply! I am sorry I thought it is appropriate in Academia, actual reviewer said that we should include more elements and test, so there are many more elements tested in the revised version than the original version, this is the another reason I felt I should move to bonferroni. However you are right that I feel "I cannot discuss this much more elements" and don't like the result, I think I stay in BH anyway. – dumm Dec 24 '19 at 2:59
  • I agree about moving this question. Now that I think about it, I was comfortable answering it because I have a PhD in statistics. – Jamie Watts Dec 27 '19 at 15:03

I'm sure you already know this because you have been working with it in your manuscript, but the Benjamini Hochberg procedure is for avoiding false discoveries. So, in other words, Benjamini Hochberg is, I'm sure you know, a procedure you use to reduce Type I errors, where you incorrectly reject true null hypotheses, which are false positives results.

If, upon revising,you discovered "many spurious results that very subtle change are considered significant," it sounds to me as though you may, possibly, have considered some of the results in your latest draft to be false positives, prompting you to change your statistical procedure to Bonferroni for your next draft.

The key point to keep in mind here is that Benjamini Hochberg avoids false positives but Bonferroni tends to produce false negatives.

Therefore, the real question is why your switching statistical procedures. If your hypotheses, your work, and your data all indicate that the results are trending toward false positives, then Bonferroni is the way to go. Otherwise, switching procedures at this point may not be, in my own opinion, the optimal way to proceed.

|improve this answer|||||

Not the answer you're looking for? Browse other questions tagged or ask your own question.