Context: At the end of the methodology section, I explain that I will apply an ANOVA to check if at least one condition is different, then I will investigate using multiple comparisons.

The reviewers argue that I am giving away some of my results. Since I mention that I will do multiple comparisons, I am indicating that indeed at least one of the conditions differs.

How can I reformulate my analysis plan (or change my write-up) to accommodate the reviewers? Alternately, what could I answer the proverbial reviewer #2 if I want to keep the same write-up?

  • 1
    Do you agree with their analysis? – Buffy Dec 9 '20 at 14:13
  • 1
    What field are you in? In my people will regularly say how they adapted their analysis based upon the data (e.g., transformed data to meet assumptions, ran post-hoc tests, etc.) – Richard Erickson Dec 9 '20 at 14:33
  • 15
    I'm confused. "Giving away" as in "spoiling the shocking twist in your story", or in the sense of "your definition of RQ2 presumes a specific outcome of RQ1"? – xLeitix Dec 9 '20 at 14:54
  • 14
    I find that comment weird. How is that an issue? Are we supposed to treat results like a secret that the reader can only find in the Results section? You gave away your results in the abstract already. Sometimes even in the title. Anyway, just write under which conditions you plan(ned) to do a post-hoc test and how you would decide which type of post-hoc test to employ. – Roland Dec 9 '20 at 15:24
  • 4
    Reviewers of what? Grant proposal, paper? I'm confused by your question overall. Maybe you can explain more, e.g. what exactly is the problem with "giving away some of the result"? – user2705196 Dec 9 '20 at 17:46

This is a ridiculous request (edit: eh, maybe not quite so ridiculous; comments suggest that my 'however' paragraph below applies in this case, but I'll leave the rest of this answer as-is). For one, even before you have run any tests you should have a plan for post hoc testing.

I would respond that it's not possible to correct this without moving statistical methodology to the results, and that the authors feel strongly about keeping this methodology to one section. I can't imagine an editor having a problem with this response, and if the reviewer feels strongly that the boundaries between sections of a paper are sacred then this speaks to their preference directly as well.

However, make sure you aren't saying things you don't need to say if it will help please the reviewer. For example, if you write something equivalent to "we do ANOVA on apple oranges and banana models" and then say "for oranges, we then did post hoc test xyz corrected for multiple comparisons", you can drop oranges from the second section, writing "if significant, we then did post hoc test xyz corrected for multiple comparisons". It doesn't matter that you ran these tests on oranges, it matters that you planned to do this post hoc testing for any significant omnibus ANOVA results, no matter which class of fruit.

  • 5
    The reviewer comment might have been motivated by the fact that I was saying stuff like "for oranges, we then did". I corrected the write-up to keep it more generic. – GuillaumeL Dec 9 '20 at 16:45
  • 3
    @GuillaumeL Ah, that makes it much easier then! Always better to be able to make a change the reviewer will agree with rather than having to argue against something that is truly silly... – Bryan Krause Dec 9 '20 at 16:47
  • In your reply to reviewer section say. "We agree with the reviewer, and therefore removed 'for oranges, we ..., and replaced it with 'for any significant results, we did a posthoc test corrected for multiple comparisons" and do not mention anything else that you disagree with. Reviewer will likely be very happy with this. Definitely don't mention if you disagree with one of the suggestions. My tip is to always only mention what you changed/agree with, unless there is strong reason to mention disagreements. – WetlabStudent Dec 10 '20 at 1:43
  • While it's true that "it matters that you planned to do this post hoc testing for any significant omnibus ANOVA results, no matter which class of fruit," it should go without saying that you shouldn't say that unless it's actually true. If you first did the ANOVA and then only after that decided to do the xyz test on oranges, then you should make that clear. – Nathaniel Dec 10 '20 at 6:12

Without seeing the reviewers' exact comments, it's hard to say, but I suspect that they may simply want to you be clear about what you decided to do before performing the experiment, and what you decided to do after performing the experiment. In "run-of-the-mill" science, we form a hypothesis, design an experiment to test the hypothesis, and then perform the experiment. The order in which these things happen is very important when interpreting the strength of the results. Usually, the "Methodology" section in a paper describes the design of the experiment: i.e. the stuff that you decided to do before performing the experiment. If you put stuff out of chronological order, then it can be difficult to determine whether you decided to do activity X before or after previewing the results.

I'm not suggesting that it is absolutely crucial that you organize your paper in this fashion, but I think that this is the crux of your reviewers' concerns. So, you can probably appease them by adding clarity surrounding the chronology of events: If you have hypotheses that you formed before the experiment and hypotheses that you formed "during" the experiment be clear about this; if you changed course in your experiment because of preliminary results - be clear about what the original plan was, what the amended plan is, when in the experiment the amendment occurred and why.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.