44

One of the reviewers of my conference paper (computer science), which got a revise & resubmit, criticized that my conclusions are not well founded in my experiment and that they don’t show my hypothesis. (In detail, there was no control variable, which he correctly remarked.)

The point is, I wanted to use this experiment to give an idea for the direction of the work following my results and to generalize my previous findings.

My question is, how could I respond to this comment, as deleting my discussion would cut out a huge part of my work?

4
  • 2
    Can you be more specific about the “work following my results” and the “previous findings” and tell us whether and where they were published? Anyway, if you have some good reasons why you want this content in your discussion, why don’t you give this reasons in the paper or at least to the reviewer?
    – Wrzlprmft
    Jul 27, 2014 at 19:41
  • The work I am doing now but cannot put in this paper yet. The previous findings build the foundation of this paper, which was published at another conference in the same research area. The reviewer didn't give any suggestions on how to improve the from him criticized part, could you then give me a phrase how to respond to this in an elegant way? Jul 27, 2014 at 19:47
  • 3
    I suggest sitting down with your advisor and sorting the feedback you received into good (as in constructive) and bad (as in worthless) piles. The good stuff you can start cranking away on. You should then prepare an "author's response to reviewer's feedback" document in which you explain in respectful but unwavering terms why you did/did not choose to incorporate the various action items stemming from the referee report. Submit this document along with your resubmission.
    – Mad Jack
    Jul 27, 2014 at 20:58
  • 2
    That's a valid risk of the 'future work' section - if the described experiments as such seem to the reviewer as insufficient, a likely reaction is "the direction is fine, but come back when you've done the next step and added it to this paper".
    – Peteris
    Jul 27, 2014 at 23:26

5 Answers 5

46

First of all, I love the title of your question. What an open, honest way of phrasing things, which makes for a question which just about every academic I know can identify with.

Whenever you get a "revise and resubmit" referee report, the two key questions are:

  • How many of the revisions do you want to make? How many can you make in a reasonable amount of time?

  • Almost certainly you will resubmit, yes, but: to the same journal or to a different journal?

I take it that by "completely correct" you mean that you agree that all of the suggested revisions would improve the paper. That still does not imply that you want to make all of them: if the revisions ask for significant further work, then it may well be that, yes, you are in the midst of this work / planning to do it in the future but nevertheless you are seeking to publish what you have done so far.

Of course if you agree with all of the suggested revisions and you feel that you can do them in a reasonable amount of time -- e.g., in time to meet the deadline for a conference -- then it seems pretty clear that's the way to go.

I take it though that in your case it is not practical to complete all the suggested revisions in the given amount of time. Your remaining choices are then: (i) complete only some of the revisions, explain very carefully in your reply why you considered the other proposed revisions but did not make them, and hope for the best; (ii) indeed resubmit, but elsewhere; and (iii) withdraw the paper until you can complete the suggested work. Then resubmit (possibly to the same place, if applicable, but starting over again in the formal submission process).

These are tough choices, and obviously they cannot be made globally. All I'll say is that the more (!!) "completely correct" you feel the reviewer's suggested changes are, the more likely it is that if you do not satisfactorily incorporate these changes then the paper will not be accepted. This still does not mean that revising and resubmitting is a poor choice: you have to do an expected value computation (e.g. if you can respond to 2/3 of the suggested revisions within a few days, maybe give resubmission to the same conference a whirl: why not?) to decide what is your best option. This decision also includes how important it is to you that the paper be published sooner rather than later and how important that it be published in this specific venue. Ideally speaking, you should take the necessary time to publish a "complete" version of your work rather than an unsatisfactorily partial preliminary version...but in reality, many academics do not have the luxury of fully indulging this ideal.

2
  • 2
    'I take it that by "completely correct" you mean that you agree that all of the suggested revisions would improve the paper.' - based on the question, I had expected exactly that not to be the case (but as the OP accepted this answer, apparently they meant it like interpreted here). Jul 28, 2014 at 9:17
  • 3
    If one does resubmit without implementing such a correction, it is important to include an explanation of this both in the direct response to the referee and in the paper itself. Reviewers are test readers, so if they caught this then some fraction of the intended readership will also go "why isn't there a control variable?", and it's important that the paper itself make it clear that that is either not possible/technically hard, or work in progress, or so on.
    – E.P.
    Jul 28, 2014 at 16:35
18

Let me first recapture your situation as I understood it and based my answer on:

  • You have essentially three Papers, let’s call them Paper A, B and C. Paper A has already been published, Paper B is the one you are currently working on and Paper C is a planned future publication.
  • You drew conclusions in Paper B, which are not based on the research of Paper B alone but also require results from Papers A and C.
  • The reviewer correctly remarked that your conclusions were not supported by your research in Paper B.

First of all, as your conclusions are not supported by published research yet, you cannot make these conclusions and there is no (ethical) way to leave them as conclusions. Rather from the situation at the end of Paper B, they are an outlook or speculations and thus should be marked as such.

E.g., you could write something along the lines of the following in your conclusions:

Our results confirm the findings of [Paper A], which suggests [general hypothesis]. This could be confirmed by [ansatz for research of Paper C].

To the reviewer’s comment you could reply along the lines of the following:

The reviewer is indeed correct that [general hypothesis] is not supported by our current findings. Rather, [general hypothesis] is something suggested by our results from our findings and [Paper A] and should be investigated in future research. We have insufficiently described this point and amended our manuscript such that this aspect is now made clear.

4
  • 5
    It sounds like the current and forthcoming papers have a circular dependency.
    – user18072
    Jul 28, 2014 at 0:00
  • 7
    There's a long tradition in some fields of basing work on a conjecture which is only later proven.
    – Ben Voigt
    Jul 28, 2014 at 6:43
  • @djechlin: At least in my scenario, and that’s exactly what I am suggesting to resolve.
    – Wrzlprmft
    Jul 28, 2014 at 14:43
  • 1
    @BenVoigt yes, and said intermediate work will use the conjecture but not rely on the yet-extant proof.
    – user18072
    Jul 28, 2014 at 14:45
10

I think the solution for the problem in the title is to revise the paper so that the undesirable consequences are no longer consequences. As I understand it, the reviewer correctly pointed out that your conclusion is not adequately supported by your experiment; the undesirable consequence is that this conclusion should be removed; it's undesirable because that would mean removing a big part of the paper. Fortunately, the undesirable consequence would no longer be a consequence at all if the conclusion were relabeled to match what you wrote in the second paragraph of your question --- for example along the lines of Wrzlprmft's suggestion. More generally, instead of saying (incorrectly) that something is a conclusion from your experiment, say what it really is, as accurately as you can.

9

My question is, how could I respond to this comment, as deleting my discussion would cut out a huge part of my work?

Trust me. I have been there.

Remove the questionable part, and even the whole paper if necessary. (Well, unless the reviewer is a well-known idiot :-) )

In the future, you will regret anything you published that is not perfectly fine and relevant.

PS. Also, in the future, you may be involved in some competition or conflictual circumstance where a "competitor" or opponent may use a "scientific weakness" against you. And in this case you may "pay" a high price.

3
  • 1
    I absolutely agree. We give them what they want, there is nothing we can do against them anyway. Whether, you like or not like the consequence is not relevant.
    – InformedA
    Jul 28, 2014 at 5:18
  • 2
    @randomA Who is "them"? The reviewers? That's also "us".
    – Raphael
    Jul 29, 2014 at 6:13
  • @Raphael Depending on how they see things. In most cases, they are very much different from 'us'.
    – InformedA
    Jul 29, 2014 at 8:01
2

Firstly, I'm not totally clear about the exact situation. Here's how I understand it:

  • some experiments have been done and analysed
  • some conclustion can be drawn
  • some control experiment that would allow far more general conclusions is planned but has not yet been done.
    This is what the reviewer is worried about.

Author point of view:

I'd say that the decision what to do depends on whether the conclusions that can be drawn are important enough to write a paper about them. Alternatively, the other way round: whether a paper written about both the already performed and the control experiments would be too long for a good paper.

In these situations, I'd

  • agree with the reviewer (in the answer to the reviewers)
  • but point out that after careful consideration you decided that the control experiments are out of the scope of the present (preliminary? feasibility? case?) study.
  • In addition, I'd spell out the limitations of the present study very clearly
  • while outlining the follow-up experiment as a solution. This needs some care in order to convince readers (and reviewer) that the follow-up experiment is really taking place: the literature is full of ongoing experiments that never went on.
  • In the answer to the reviewers, state explicitly that you took care to clearly outline the limitatins of the present study due to the lack of control experiments.

Obviously, this approach can only work if the lack of control experiments has some sensible reason and doesn't appear to be lazyness. Spell out the reasons (careful with non-scientific reasons like feasibility [vs. laziness]).


Reviewer point of view

When reviewing a paper, I find it quite OK if experiments have limitations* if the authors are clearly aware of these limitations. I tend to be worried if the conclusion that the world is rescued is based on 3 dead mice.

* Experiments will always have limitations. It is IMHO perfectly right to look at science as an iterative process of refining knowledge. It doesn't make sense to get thousands of patients for a preliminary study. But the preliminary study should spell out the limitations. And in practice, the preliminary study is (and should be) needed in order to get the thousands of patients, and/or better meta information, or other other labs joining the efforts etc. for follow-up studies. But also this will work only if the preliminary study is honest about limitations. Otherwise the topic will be perceived as "solved".

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .