5

Context: At the end of the methodology section, I explain that I will apply an ANOVA to check if at least one condition is different, then I will investigate using multiple comparisons.

The reviewers argue that I am giving away some of my results. Since I mention that I will do multiple comparisons, I am indicating that indeed at least one of the conditions differs.

How can I reformulate my analysis plan (or change my write-up) to accommodate the reviewers? Alternately, what could I answer the proverbial reviewer #2 if I want to keep the same write-up?

6
  • 1
    Do you agree with their analysis?
    – Buffy
    Dec 9, 2020 at 14:13
  • 1
    What field are you in? In my people will regularly say how they adapted their analysis based upon the data (e.g., transformed data to meet assumptions, ran post-hoc tests, etc.) Dec 9, 2020 at 14:33
  • 15
    I'm confused. "Giving away" as in "spoiling the shocking twist in your story", or in the sense of "your definition of RQ2 presumes a specific outcome of RQ1"?
    – xLeitix
    Dec 9, 2020 at 14:54
  • 14
    I find that comment weird. How is that an issue? Are we supposed to treat results like a secret that the reader can only find in the Results section? You gave away your results in the abstract already. Sometimes even in the title. Anyway, just write under which conditions you plan(ned) to do a post-hoc test and how you would decide which type of post-hoc test to employ.
    – user9482
    Dec 9, 2020 at 15:24
  • 4
    Reviewers of what? Grant proposal, paper? I'm confused by your question overall. Maybe you can explain more, e.g. what exactly is the problem with "giving away some of the result"? Dec 9, 2020 at 17:46

2 Answers 2

15

This is a ridiculous request (edit: eh, maybe not quite so ridiculous; comments suggest that my 'however' paragraph below applies in this case, but I'll leave the rest of this answer as-is). For one, even before you have run any tests you should have a plan for post hoc testing.

I would respond that it's not possible to correct this without moving statistical methodology to the results, and that the authors feel strongly about keeping this methodology to one section. I can't imagine an editor having a problem with this response, and if the reviewer feels strongly that the boundaries between sections of a paper are sacred then this speaks to their preference directly as well.

However, make sure you aren't saying things you don't need to say if it will help please the reviewer. For example, if you write something equivalent to "we do ANOVA on apple oranges and banana models" and then say "for oranges, we then did post hoc test xyz corrected for multiple comparisons", you can drop oranges from the second section, writing "if significant, we then did post hoc test xyz corrected for multiple comparisons". It doesn't matter that you ran these tests on oranges, it matters that you planned to do this post hoc testing for any significant omnibus ANOVA results, no matter which class of fruit.

4
  • 5
    The reviewer comment might have been motivated by the fact that I was saying stuff like "for oranges, we then did". I corrected the write-up to keep it more generic.
    – GuillaumeL
    Dec 9, 2020 at 16:45
  • 3
    @GuillaumeL Ah, that makes it much easier then! Always better to be able to make a change the reviewer will agree with rather than having to argue against something that is truly silly...
    – Bryan Krause
    Dec 9, 2020 at 16:47
  • In your reply to reviewer section say. "We agree with the reviewer, and therefore removed 'for oranges, we ..., and replaced it with 'for any significant results, we did a posthoc test corrected for multiple comparisons" and do not mention anything else that you disagree with. Reviewer will likely be very happy with this. Definitely don't mention if you disagree with one of the suggestions. My tip is to always only mention what you changed/agree with, unless there is strong reason to mention disagreements. Dec 10, 2020 at 1:43
  • While it's true that "it matters that you planned to do this post hoc testing for any significant omnibus ANOVA results, no matter which class of fruit," it should go without saying that you shouldn't say that unless it's actually true. If you first did the ANOVA and then only after that decided to do the xyz test on oranges, then you should make that clear.
    – N. Virgo
    Dec 10, 2020 at 6:12
4

Without seeing the reviewers' exact comments, it's hard to say, but I suspect that they may simply want to you be clear about what you decided to do before performing the experiment, and what you decided to do after performing the experiment. In "run-of-the-mill" science, we form a hypothesis, design an experiment to test the hypothesis, and then perform the experiment. The order in which these things happen is very important when interpreting the strength of the results. Usually, the "Methodology" section in a paper describes the design of the experiment: i.e. the stuff that you decided to do before performing the experiment. If you put stuff out of chronological order, then it can be difficult to determine whether you decided to do activity X before or after previewing the results.

I'm not suggesting that it is absolutely crucial that you organize your paper in this fashion, but I think that this is the crux of your reviewers' concerns. So, you can probably appease them by adding clarity surrounding the chronology of events: If you have hypotheses that you formed before the experiment and hypotheses that you formed "during" the experiment be clear about this; if you changed course in your experiment because of preliminary results - be clear about what the original plan was, what the amended plan is, when in the experiment the amendment occurred and why.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .