How to avoid the "you didn't consider this" review in a scientific publication?

Question

I am a 5th year PhD student, and I have been writing scientific papers for conferences for a few years now. One constant in the feedback that we get from reviewers is what I call, for lack of a better definition, the "you didn't consider this" comment.

I am working on an emerging technology, let's call it E, that has certain advantages with respect to the industry standard, let's call it S. E has of course its disadvantages too, and my job as a researcher is trying to address the major shortcomings, let's call them E.a, E.b, E.c, and E.d.

The problem that I am facing is that, when I write a paper addressing E.a, a common feedback that I get is "you have good results, but what about E.b? There is no way this is going to work when you consider E.c and E.d as well".

While I understand the concern of the reviewers, I am also frustrated by the fact that a paper is rejected not because my work on E.a is poor, but because there is a lack of research and solutions to address E.b, E.c, and E.d. Moreover, I don't know how to improve a paper that was rejected based on those reviews, because they are not specific about the content of the paper.

My question is: how can I present my work in a way that makes it clear that only E.a is addressed, and minimize the chances of getting those unhelpful reviews? Is it possible to be explicit about it without being unbecoming?

A few notes:

1. I am concerned about conferences, so paper are limited to 4 to 8 pages, and this limits the amount of background that can be covered in the introduction,
2. Because I am addressing an emerging technology, I usually have at least one page introducing the basic principles of it before moving on to my contribution on E.a,
3. I do mentioned E.b, E.c, and E.d and cite relevant research in that regard.

Answer 1

I empathise with your difficulty and wish there was an easy fix. However, going by the question and comments, it does seem like you may need to rethink your approach a little. Consider the following:

(1) In an applied research field, the bar is not just solving a sub-problem; it is ensuring that the proposed solution will work with all other factors considered. To give a very trivial and crude (pardon me) example, if a car bonnet has to be light and strong, you cannot treat the two independently and use polythene (solving light) or concrete (solving strong).

The comment : "There is no way this is going to work when you consider E.c and E.d as well"." seems to suggest exactly that. I know my example is not perfectly analogous to your problem, but I hope the essential message is conveyed.

(2) There exists an industry standard S. Therefore, any proposal to change has merit only if it can replace S on all counts. If the method S scores 70%, 80%, 60% on some relevant metrics, your method E should not score 100%, 10%, 80% , even though 2/3 metrics are significantly improved (because one metric has fallen, and that could be unacceptable). Rather, an incremental 75%, 85%, 65% would be more acceptable.

There is a constant strain in such fields between novelty and applicability, both need to be balanced out. Depending on what conference it is, one may carry more weight than the other, but the application bit can rarely be ignored.

What seems to me a better approach is:

(1) Tackle Eb/c/d in your study, even if briefly. Not just in the introduction, in the study and the results itself. Show that you are not simply highlighting problems (i.e. mentioning only in introduction), but doing something about them even if you don't solve them fully. Even analysing and rating the magnitude/importance of Eb/c/d is a meaningful contribution, supplementing your main work on Ea. So you don't need to solve them all, but you do need to consider them and look at inter-dependence.

(2) If possible, suggest ways in which E can be integrated with S. Then, the benefits of Ea can be used without the problems Eb, Ec. Again, you don't actually need to integrate the whole method. Look at ways to do it, what problems exist, how meaningful such an exercise is vs replacing an entire system.

These approaches should be good for such conferences which are indeed intended to share a developing idea. Even in those, your eye must be on the future- on how to develop it, what can be taken forward, what can be done to make it more complete.

Answer 2

Aside from working harder or more cleverly at pleasing these reviewers, there are two major "solutions" that come to mind, which I think are important to talk about:

1. Read the Conference or Journal you submit to. A lot: This will make you understand what is needed in your analysis, how to sidestep the incomplete/unsolved aspects of your method, and generally how to frame your results in a way the reviewers will like. You need to change your mindset from the student who reads a paper to expand your knowledge, to a reviewer mindset who reads a paper to understand where it fits in the field, and whether it's worth caring about (out of necessity, because that's the only way you'll be able to read large numbers of papers). This will also help you see how to make your case more concisely by using the right jargon and citations, and better fit what you need into the short conference paper.

2. Work on the open problems at the cutting edge of your field: this is something a person who isn't in the "hot" areas won't want to hear I'm sure. And I don't think it's necessarily right or fair either. But you asked a hard question. The solution is going to be a doozie. If you're solving mature and "solved" problems (but better) it puts a big target on your work for a range of reasons, some good and some not. Reviewers will start from a mindset of "how can I quickly shoot down this paper I don't want to read". At the very least, this conference may not be the best venue. The task of convincing people your problem is really unsolved and your approach is important, may be one too many things to cram into a short conference paper. You are up against other papers that don't need to work nearly as hard to make that case and thus have more room for results.

Now if the above doesn't apply to you because you're already doing it, you may just have an unlucky streak. There's a lot of bad reviewers out there, frustrating us all. Plus with acceptance rates down in the teens and below, plenty of excellent papers are getting rejected.

Answer 3

There is simply no way to be sure that these question will arise. You may write more references, but as space is limited, you will have to make some choices.

Consider those feedbacks as help: when a paper is accepted and you present it at a conference, those questions may arise afterwards in Q&A session. Be prepared for that, as well as take them into account in the PhD thesis.

Answer 4

That sucks - it sounds like you are already doing what you can reasonably do in the word limit of the paper. I would suggest re-reading your introduction and background sections to see if you can stress the background and scope any more clearly, to head off this objection. (Really beat the reader over the head with it.) If this part is already clear, then respond to these referee reports (for a revise-and-resubmit) by pointing out the background they may have missed. If it is an outright rejection, try another journal.

Remember that you have to respond to all referee comments when re-submitting, but you do not have to agree. If the comment is not sensible, tell them that you are not making any change and say why. If your reasons are good, the editor might decide that the objection made by the referee is not helpful. Here is an example.

Referee comment: The author's work on E.a is interesting, but he has failed to address problems with E.b and E.c. Without this, I do not see how his method could be a viable alternative to S. The author should expand his discussion to explain how to deal with E.b and E.c, and explain how these problems (and their solutions) interact.

Response (Disagree - No change): The present paper is confined in scope to solving problem E.a, though the problems mentioned by the referee are important, and are acknowledged in the background section (p. 2). The introduction and background sections are clear on the limited scope of my paper. The space limits for this paper do not allow for the broader argument suggested by the referee. I note that these other problems are an avenue of current research that I am pursuing in another paper, and it is my hope that this will eventually culminate in a set of papers that solve all problems for method E, with a summary paper to tie them together. Nevertheless, the present paper, with the imposed word limit, is not the place for this broader research. I have re-read my background section and I am satisfied that the present paper is clear and reasonable in its scope, and no further change is required on this point.