How to react to a reviewer’s comment when he is completely right, but you do not like the consequences?

Question

One of the reviewers of my conference paper (computer science), which got a revise & resubmit, criticized that my conclusions are not well founded in my experiment and that they don’t show my hypothesis. (In detail, there was no control variable, which he correctly remarked.)

The point is, I wanted to use this experiment to give an idea for the direction of the work following my results and to generalize my previous findings.

My question is, how could I respond to this comment, as deleting my discussion would cut out a huge part of my work?

Answer 1

First of all, I love the title of your question. What an open, honest way of phrasing things, which makes for a question which just about every academic I know can identify with.

Whenever you get a "revise and resubmit" referee report, the two key questions are:

• How many of the revisions do you want to make? How many can you make in a reasonable amount of time?

• Almost certainly you will resubmit, yes, but: to the same journal or to a different journal?

I take it that by "completely correct" you mean that you agree that all of the suggested revisions would improve the paper. That still does not imply that you want to make all of them: if the revisions ask for significant further work, then it may well be that, yes, you are in the midst of this work / planning to do it in the future but nevertheless you are seeking to publish what you have done so far.

Of course if you agree with all of the suggested revisions and you feel that you can do them in a reasonable amount of time -- e.g., in time to meet the deadline for a conference -- then it seems pretty clear that's the way to go.

I take it though that in your case it is not practical to complete all the suggested revisions in the given amount of time. Your remaining choices are then: (i) complete only some of the revisions, explain very carefully in your reply why you considered the other proposed revisions but did not make them, and hope for the best; (ii) indeed resubmit, but elsewhere; and (iii) withdraw the paper until you can complete the suggested work. Then resubmit (possibly to the same place, if applicable, but starting over again in the formal submission process).

These are tough choices, and obviously they cannot be made globally. All I'll say is that the more (!!) "completely correct" you feel the reviewer's suggested changes are, the more likely it is that if you do not satisfactorily incorporate these changes then the paper will not be accepted. This still does not mean that revising and resubmitting is a poor choice: you have to do an expected value computation (e.g. if you can respond to 2/3 of the suggested revisions within a few days, maybe give resubmission to the same conference a whirl: why not?) to decide what is your best option. This decision also includes how important it is to you that the paper be published sooner rather than later and how important that it be published in this specific venue. Ideally speaking, you should take the necessary time to publish a "complete" version of your work rather than an unsatisfactorily partial preliminary version...but in reality, many academics do not have the luxury of fully indulging this ideal.

Answer 2

Let me first recapture your situation as I understood it and based my answer on:

• You have essentially three Papers, let’s call them Paper A, B and C. Paper A has already been published, Paper B is the one you are currently working on and Paper C is a planned future publication.
• You drew conclusions in Paper B, which are not based on the research of Paper B alone but also require results from Papers A and C.
• The reviewer correctly remarked that your conclusions were not supported by your research in Paper B.

First of all, as your conclusions are not supported by published research yet, you cannot make these conclusions and there is no (ethical) way to leave them as conclusions. Rather from the situation at the end of Paper B, they are an outlook or speculations and thus should be marked as such.

E.g., you could write something along the lines of the following in your conclusions:

Our results confirm the findings of [Paper A], which suggests [general hypothesis]. This could be confirmed by [ansatz for research of Paper C].

To the reviewer’s comment you could reply along the lines of the following:

The reviewer is indeed correct that [general hypothesis] is not supported by our current findings. Rather, [general hypothesis] is something suggested by our results from our findings and [Paper A] and should be investigated in future research. We have insufficiently described this point and amended our manuscript such that this aspect is now made clear.

Answer 3

I think the solution for the problem in the title is to revise the paper so that the undesirable consequences are no longer consequences. As I understand it, the reviewer correctly pointed out that your conclusion is not adequately supported by your experiment; the undesirable consequence is that this conclusion should be removed; it's undesirable because that would mean removing a big part of the paper. Fortunately, the undesirable consequence would no longer be a consequence at all if the conclusion were relabeled to match what you wrote in the second paragraph of your question --- for example along the lines of Wrzlprmft's suggestion. More generally, instead of saying (incorrectly) that something is a conclusion from your experiment, say what it really is, as accurately as you can.

Answer 4

My question is, how could I respond to this comment, as deleting my discussion would cut out a huge part of my work?

Trust me. I have been there.

Remove the questionable part, and even the whole paper if necessary. (Well, unless the reviewer is a well-known idiot :-) )

In the future, you will regret anything you published that is not perfectly fine and relevant.

PS. Also, in the future, you may be involved in some competition or conflictual circumstance where a "competitor" or opponent may use a "scientific weakness" against you. And in this case you may "pay" a high price.

Answer 5

Firstly, I'm not totally clear about the exact situation. Here's how I understand it:

• some experiments have been done and analysed
• some conclustion can be drawn
• some control experiment that would allow far more general conclusions is planned but has not yet been done.
  This is what the reviewer is worried about.

Author point of view:

I'd say that the decision what to do depends on whether the conclusions that can be drawn are important enough to write a paper about them. Alternatively, the other way round: whether a paper written about both the already performed and the control experiments would be too long for a good paper.

In these situations, I'd

• agree with the reviewer (in the answer to the reviewers)
• but point out that after careful consideration you decided that the control experiments are out of the scope of the present (preliminary? feasibility? case?) study.
• In addition, I'd spell out the limitations of the present study very clearly
• while outlining the follow-up experiment as a solution. This needs some care in order to convince readers (and reviewer) that the follow-up experiment is really taking place: the literature is full of ongoing experiments that never went on.
• In the answer to the reviewers, state explicitly that you took care to clearly outline the limitatins of the present study due to the lack of control experiments.

Obviously, this approach can only work if the lack of control experiments has some sensible reason and doesn't appear to be lazyness. Spell out the reasons (careful with non-scientific reasons like feasibility [vs. laziness]).

---

Reviewer point of view

When reviewing a paper, I find it quite OK if experiments have limitations* if the authors are clearly aware of these limitations. I tend to be worried if the conclusion that the world is rescued is based on 3 dead mice.

* Experiments will always have limitations. It is IMHO perfectly right to look at science as an iterative process of refining knowledge. It doesn't make sense to get thousands of patients for a preliminary study. But the preliminary study should spell out the limitations. And in practice, the preliminary study is (and should be) needed in order to get the thousands of patients, and/or better meta information, or other other labs joining the efforts etc. for follow-up studies. But also this will work only if the preliminary study is honest about limitations. Otherwise the topic will be perceived as "solved".