8

I am a 5th year PhD student, and I have been writing scientific papers for conferences for a few years now. One constant in the feedback that we get from reviewers is what I call, for lack of a better definition, the "you didn't consider this" comment.

I am working on an emerging technology, let's call it E, that has certain advantages with respect to the industry standard, let's call it S. E has of course its disadvantages too, and my job as a researcher is trying to address the major shortcomings, let's call them E.a, E.b, E.c, and E.d.

The problem that I am facing is that, when I write a paper addressing E.a, a common feedback that I get is "you have good results, but what about E.b? There is no way this is going to work when you consider E.c and E.d as well".

While I understand the concern of the reviewers, I am also frustrated by the fact that a paper is rejected not because my work on E.a is poor, but because there is a lack of research and solutions to address E.b, E.c, and E.d. Moreover, I don't know how to improve a paper that was rejected based on those reviews, because they are not specific about the content of the paper.

My question is: how can I present my work in a way that makes it clear that only E.a is addressed, and minimize the chances of getting those unhelpful reviews? Is it possible to be explicit about it without being unbecoming?

A few notes:

  1. I am concerned about conferences, so paper are limited to 4 to 8 pages, and this limits the amount of background that can be covered in the introduction,
  2. Because I am addressing an emerging technology, I usually have at least one page introducing the basic principles of it before moving on to my contribution on E.a,
  3. I do mentioned E.b, E.c, and E.d and cite relevant research in that regard.
|improve this question
  • 1
    What answer are you looking for beyond the obvious "write more exposition"? It sounds like you already know the answer but you don't like it. – user9646 May 10 '18 at 18:30
  • 2
    I think the issue maybe that reviewers think that E.a (or one of the others) is the 'easy' problem; i.e., the solution is obvious or trivial, and thus is not an interesting case. If this is true, then you need to pick the hardest possible problem; this may be the case when you combine E.a to E.d (joint problem). Once you pick the hardest problem, then you can just say these other sub-problems can be easily addressed via method-1, method-2, etc. – Prof. Santa Claus May 11 '18 at 7:52
  • 2
    Sometimes these comments mean "You did not cite my paper on E.b." All you can do is try to cite them. Submit papers places that allow revisions. – Anonymous Physicist May 11 '18 at 8:33
  • @NajibIdrissi: That's fair, and the reason why I don't like the obvious answer is that it doesn't seem to work. Perhaps I have to improve on how I explain the problem statement. – Leo May 11 '18 at 16:30
  • 1
    @Prof.SantaClaus: That's an interesting point. But if I tackle the hardest problem which is made up by several sub-problems, why would I not submit that to a journal? My understanding is that, at least in electrical engineering, one submits simpler work to a conference, and then a much extended version of a project that is based on 2/3 conference papers. – Leo May 11 '18 at 16:34
3

I empathise with your difficulty and wish there was an easy fix. However, going by the question and comments, it does seem like you may need to rethink your approach a little. Consider the following:

(1) In an applied research field, the bar is not just solving a sub-problem; it is ensuring that the proposed solution will work with all other factors considered. To give a very trivial and crude (pardon me) example, if a car bonnet has to be light and strong, you cannot treat the two independently and use polythene (solving light) or concrete (solving strong).

The comment : "There is no way this is going to work when you consider E.c and E.d as well"." seems to suggest exactly that. I know my example is not perfectly analogous to your problem, but I hope the essential message is conveyed.

(2) There exists an industry standard S. Therefore, any proposal to change has merit only if it can replace S on all counts. If the method S scores 70%, 80%, 60% on some relevant metrics, your method E should not score 100%, 10%, 80% , even though 2/3 metrics are significantly improved (because one metric has fallen, and that could be unacceptable). Rather, an incremental 75%, 85%, 65% would be more acceptable.

There is a constant strain in such fields between novelty and applicability, both need to be balanced out. Depending on what conference it is, one may carry more weight than the other, but the application bit can rarely be ignored.

What seems to me a better approach is:

(1) Tackle Eb/c/d in your study, even if briefly. Not just in the introduction, in the study and the results itself. Show that you are not simply highlighting problems (i.e. mentioning only in introduction), but doing something about them even if you don't solve them fully. Even analysing and rating the magnitude/importance of Eb/c/d is a meaningful contribution, supplementing your main work on Ea. So you don't need to solve them all, but you do need to consider them and look at inter-dependence.

(2) If possible, suggest ways in which E can be integrated with S. Then, the benefits of Ea can be used without the problems Eb, Ec. Again, you don't actually need to integrate the whole method. Look at ways to do it, what problems exist, how meaningful such an exercise is vs replacing an entire system.

These approaches should be good for such conferences which are indeed intended to share a developing idea. Even in those, your eye must be on the future- on how to develop it, what can be taken forward, what can be done to make it more complete.

|improve this answer
  • With respect, I think this is bad advice. You seem to be suggesting that it is illegitimate to confine the scope of a paper to solving one single problem, and that the only solution is to bend to the requirement to solve the whole set of problems in one paper. In my opinion, attempting a "brief" (and therefore superficial) "tackle" of the other problems is worse than leaving them out-of-scope. – Reinstate Monica May 11 '18 at 23:48
  • "If the method S scores 70%, 80%, 60% on some relevant metrics, your method E should not score 100%, 10%, 80%." I disagree. The industry standard is standard for a reason: it's been thoroughly tested and implemented. You cannot expect a prototype or novel idea to score better on every metric. – FBolst May 11 '18 at 23:59
  • @Ben - Illegitimate is an unnecessarily strong word; it is unlikely to pass muster IF the problems are inter-related. When the problems are independent, one can always provide a suitable rebuttal. I disagree that brief=superficial; it's a strange equivalence that suggests short papers/letters are superficial compared to longer papers. The idea is to take everything in perspective, and with that view, focus on one problem. As against turning a blind eye to other problems that the suggested change will cause (as reviewers seem to suggest). You are of course, entitled to your opinion. :) – AppliedAcademic May 12 '18 at 0:37
  • @FBolst- precisely why industry is unlikely to accept it as a potential result, atleast until that 10% can be raised reasonably. That's why I said the eye should be on the future-how serious is the 10% metric, how can it be raised? Is there an application where this 10% is not so relevant and the other metrics carry more weight? If these are not addressed, the first question, from personal experience, is "what's the point of this?". It's harsh to hear, but that's a fact of the field. – AppliedAcademic May 12 '18 at 0:54
  • 1
    @user153812 - Thank you for the answer, I think it sums up the generic reviewer's feelings about it. In particular I think your advice to solve (1) is very good, in the way that is stresses throughout the paper the fact that the other subproblems are being addressed somewhere else in the literature. It's frustrating to realize that the "divide and conquer" approach is so frowned upon in this respect, but at the same time I realized that I tend to behave that way too when I help my advisor to review papers. Receiving such reviews made me, incidentally, a better reviewer! – Leo May 24 '18 at 22:14
2

Aside from working harder or more cleverly at pleasing these reviewers, there are two major "solutions" that come to mind, which I think are important to talk about:

1. Read the Conference or Journal you submit to. A lot: This will make you understand what is needed in your analysis, how to sidestep the incomplete/unsolved aspects of your method, and generally how to frame your results in a way the reviewers will like. You need to change your mindset from the student who reads a paper to expand your knowledge, to a reviewer mindset who reads a paper to understand where it fits in the field, and whether it's worth caring about (out of necessity, because that's the only way you'll be able to read large numbers of papers). This will also help you see how to make your case more concisely by using the right jargon and citations, and better fit what you need into the short conference paper.

2. Work on the open problems at the cutting edge of your field: this is something a person who isn't in the "hot" areas won't want to hear I'm sure. And I don't think it's necessarily right or fair either. But you asked a hard question. The solution is going to be a doozie. If you're solving mature and "solved" problems (but better) it puts a big target on your work for a range of reasons, some good and some not. Reviewers will start from a mindset of "how can I quickly shoot down this paper I don't want to read". At the very least, this conference may not be the best venue. The task of convincing people your problem is really unsolved and your approach is important, may be one too many things to cram into a short conference paper. You are up against other papers that don't need to work nearly as hard to make that case and thus have more room for results.

Now if the above doesn't apply to you because you're already doing it, you may just have an unlucky streak. There's a lot of bad reviewers out there, frustrating us all. Plus with acceptance rates down in the teens and below, plenty of excellent papers are getting rejected.

|improve this answer
  • 'You are up against other papers that don't need to work nearly as hard to make that case and thus have more room for results.' - well said! Another harsh reality. – AppliedAcademic May 12 '18 at 0:56
1

There is simply no way to be sure that these question will arise. You may write more references, but as space is limited, you will have to make some choices.

Consider those feedbacks as help: when a paper is accepted and you present it at a conference, those questions may arise afterwards in Q&A session. Be prepared for that, as well as take them into account in the PhD thesis.

|improve this answer
1

That sucks - it sounds like you are already doing what you can reasonably do in the word limit of the paper. I would suggest re-reading your introduction and background sections to see if you can stress the background and scope any more clearly, to head off this objection. (Really beat the reader over the head with it.) If this part is already clear, then respond to these referee reports (for a revise-and-resubmit) by pointing out the background they may have missed. If it is an outright rejection, try another journal.

Remember that you have to respond to all referee comments when re-submitting, but you do not have to agree. If the comment is not sensible, tell them that you are not making any change and say why. If your reasons are good, the editor might decide that the objection made by the referee is not helpful. Here is an example.

Referee comment: The author's work on E.a is interesting, but he has failed to address problems with E.b and E.c. Without this, I do not see how his method could be a viable alternative to S. The author should expand his discussion to explain how to deal with E.b and E.c, and explain how these problems (and their solutions) interact.

Response (Disagree - No change): The present paper is confined in scope to solving problem E.a, though the problems mentioned by the referee are important, and are acknowledged in the background section (p. 2). The introduction and background sections are clear on the limited scope of my paper. The space limits for this paper do not allow for the broader argument suggested by the referee. I note that these other problems are an avenue of current research that I am pursuing in another paper, and it is my hope that this will eventually culminate in a set of papers that solve all problems for method E, with a summary paper to tie them together. Nevertheless, the present paper, with the imposed word limit, is not the place for this broader research. I have re-read my background section and I am satisfied that the present paper is clear and reasonable in its scope, and no further change is required on this point.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.