24

As this has happened to me several times for different papers now, I am not describing a specific case here, but the abstract aspects that were common to all situations.

Occasionally, in the "Tasks" or "Materials" section of the "Evaluation" or "User Study" chapter, papers of mine contain statements such as:

  • "Figure ... shows one of the tasks from the user study. The correct answer was ... ."
  • "For the user study, we had prepared eight tasks. They were given in the form of natural-language questions. Users were supposed to use our novel concept to input the appropriate parameters as described by the questions. Questions ranged from simple (such as '...?', see Figure ... for the solution) to complex (such as '...?', cf. solution in Figure ...)."

In these cases, the figure captions once again mention the figures depict questions and/or solutions from the user study.

Occasionally, these are met with reviewer comments such as:

  • "The paper does not provide any example of the tasks presented during the user study. It would greatly improve the quality of the paper if the authors could at least provide one such example. Without this, I find it very hard to get a clear idea of how the study was conducted, and thereby, I also cannot tell how reliable the results presented in Section ... are."

I base my reasoning on the idea that this reviewer comment is, plain and simple, incorrect, as I did provide concrete examples of tasks. If I am somehow mistaken about this, please do let me know.

While such statements were certainly not the only reason for voting to reject, some of the reviewers who wrote something along these lines made it rather clear that they saw the alleged lack of any task examples as one of the most critical issues with the submission.

My question focuses on the situation that such a paper is getting revised - either, because it was accepted after all, and I am preparing a camera-ready version, or because it was rejected, and I am trying to improve it before submitting it elsewhere: What is the appropriate reaction with respect to such comments?

  • Ignore the statements and do not change anything about the respective examples in my paper. This might be a viable way to go, but it leaves a bad taste to completely ignore some reviewer statements, even though they are clearly comprehensible and should be straightforward to implement.
  • Re-word the sentences mentioning the examples. I might do that, but I am wary of making things worse, as I already consider the respective statements in my paper quite unambiguous as they are.
  • Add more examples, and possibly at different locations. This could make it harder to miss the examples, but in CS conference papers, space is an extremely scarce resource.
  • Ask the editor. Especially in cases where the paper was eventually still accepted, this seems quite extreme to me. I feel contacting the editor should be reserved to truly exceptional and severe issues, not "every-day worries" of all authors who are preparing their CR-versions.

Note that I am not asking how to deal with the review decision. Even though it is slightly annoying if a rejection is indeed based on incorrect reviewer statements, I am ok with resubmitting a revised version elsewhere. I am specifically asking how to revise the particular parts that were incorrectly criticized.

Some peculiarities of my field to note that might not be obvious:

  • Only some conferences have a rebuttal phase. If they do, it is not guaranteed (or even reasonably likely) that especially reviewers who criticized a paper a lot will actually read the rebuttal and possibly even amend their review. Also, these rebuttals are usually limited in length (e.g. to 500 words), so you can usually mention only a selection of all issues raised by reviewers.
  • CR-versions are rarely accompanied with explanatory letters to the editors (or, more precisely, the program committee) in my experience.
    • Submission usually happens by means of web-based submission systems which allow the upload of exactly one PDF file, which is sometimes automatically checked on formal factors, such as number of pages (so even tacking an extra page with explanatory notes in front of the paper will not work).
    • If an opportunity is provided to upload some additional files, this is usually meant for uploading sources and supplementary material such as videos that will be put on the proceedings CD. It is not clear whether the material will be sifted thoroughly enough so letters to the editors will be found in time and treated appropriately.
  • My field is related to HCI.
    • Therefore, contributions are sometimes graphical in nature. Consequently, figures sometimes do not just provide information about the contribution (statistical graphs, exemplary architectures), figures are often a crucial part for showing the contribution. In other words, sometimes, the depiction is the contribution, or it shows how the novel concept presented in the paper does something.
    • User studies are used to test how well human users can get along with the novel concept. Therefore, tasks are often written as questions. Study participants then have to find the correct answers to these questions by using the novel concept being tested. This usually means that they have to interact/make inputs with/in the implemented prototype that shows the novel concept. This is meant to mimick how the concept would be used in real life, where users would use the respective novel HCI technique for retrieving information, as well.
  • 3
    It seems like your situation very much hangs on your specific CS discipline: strict page limits, single round review cycles, rare rebuttals. A lot of things that would come to my mind in other disciplines won't work for you. – S. Kolassa - Reinstate Monica Jul 22 '15 at 9:00
  • 2
    I assume you are in CS (me too). I am not sure about your subdiscipline, though. Because the examples you use "Figure ... shows one of the tasks from the user study. The correct answer was" or "For the user study, we had prepared eight tasks". Those examples do not sound like the typical CS paper. – Alexandros Jul 22 '15 at 9:01
  • @Alexandros: My field is HCI-related, so contributions, concepts, and/or parts of user studies are typically graphical in nature. As I think I once mentioned some time before in some other post: In my field, some figures do not merely describe data about the contribution. Some figures show, or are the contribution. – O. R. Mapper Jul 22 '15 at 9:03
  • As a casual reader, not familiar with your field, I find your example sentences a bit confusing. For instance, I find it difficult to associate a correct answer to a task, or to consider a task given in the form of questions. I also find it unclear the sentence "Users were supposed to use our novel concept to input the appropriate parameters as described by the questions". Maybe, the reviewers shared my same perplexities. – Massimo Ortolano Jul 22 '15 at 22:32
  • @MassimoOrtolano: That is an interesting remark. I have expanded the last paragraph in my question to point out the usual way user studies are done in my field. Hope that clarifies it for you. (That notwithstanding, my user study sections in the papers usually provide some more introductory information, such as "In the user study, users were given a number of questions. Each of the questions constituted a task, and users were asked to provide a correct answer.") – O. R. Mapper Jul 23 '15 at 6:45
53

I almost never ignore a reviewer comment. Even if the reviewer is simply wrong, the way in which they are wrong usually points out a way that some significant subpopulation of readers is likely to misread the paper.

Given this, when a reviewer seems to have simply overlooked a chunk of the paper, I will generally fiddle with the wording, the structure or the introductory material in order to make it pop out more clearly.

For example, with your "eight tasks" example, I might do any or all of the following to make the presence and visibility of the material more blatant:

  • Make sure that the introduction has a "the structure of this paper" subsection that includes the word "example" in it.
  • Do the same for the section including the tasks with a "structure of this section" paragraph at its beginning.
  • Add a subsection header for "Examples of Tasks"
  • Put the question examples into a bullet list, rather than prose

Remember, the reviewer may be incompetent and sloppy, but many of your readers will be also. You want to get your point across to as many of them as possible, regardless.

  • 3
    +1 for "the way in which they are wrong usually points out a way that some significant subpopulation of readers is likely to misread the paper" – mhwombat Apr 20 '16 at 10:27
  • +1 for same! This is a guiding light, even without reviewers. If your friend reads the paper and something is not clear to them, they're not going to be the only one. – reas0n Jun 2 '16 at 19:36
54

A simple rule of thumb can help to handle reviewer comments: "The reviewers are always right, even when they are wrong."

This means that when reviewers make wrong assumptions or draw wrong conclusions, apparently the article is not clear enough and allowed them to do so.

Mostly, in my experience, the solution that works best, is to restructure the article to prevent readers from drawing wrong conclusions. Make the text clearer, improve the structure, avoid being overly verbose.

The reviewers will appreciate such an approach more than you trying to point out in the reply-to-the-reviewers why they are wrong, or just ignoring their comments.

When your article was rejected immediately, then use this approach to improve the paper for a new submission, and prevent running into the same issues.

  • 11
    Could you expand your answer with a concrete example, please? When there is a chapter "User Study" with a sub-chapter "Tasks" that contains the statement "There were eight tasks, for example: '<task>'", and the conclusion by a reviewer is "No examples of tasks from the user study were provided.", I truly do not understand how I can improve the structure or make the text clearer. I would be interested in seeing how other authors handle this, without risking that at the same time, the text gets less clear for the other reviewers who found all the information. – O. R. Mapper Jul 22 '15 at 12:01
  • 3
    @O.R.Mapper, one thing you can do is scatter more pointers to the example(s) throughout the text. That works especially well if, as your original question suggests, a task is exemplified in a figure. The figure belongs to a specific section of the text, something like "methods", but already in the introduction you likely say something about the fact that you had subjects who performed tasks, and there you write in parentheses, "for examples see Sec. xy and Fig. z". – A. Donda Jul 22 '15 at 14:42
  • 3
    @O.R.Mapper: it is difficult to provide advice for your concrete paper, since I did not read your paper, nor the reviews. It might be that the specific chapter is perfectly fine, but that the overall paper structure can be improved, or that the paper is too lengthy, or that the paper deals with too many details, or that the audience of the journal and the paper do not match well, or something completely different. I understand that it is frustrating when reviewers comment about something that you consider to be completely evident (trust me, I have been there). – Danny Ruijters Jul 22 '15 at 16:46
  • 2
    Alternatively, it may be that the concrete nature of your examples are not as concrete as you may think they are - try to re-enforce the substantiality of your evidence, and highlight it as plainly as possible. – Zibbobz Jul 23 '15 at 17:12
  • @Zibbobz: "concrete" is meant as "literal" - the exact words that were shown to study participants were also written in the paper. – O. R. Mapper Jul 27 '15 at 9:31
6

The following is predicated on the assumption that the reviewer comments indeed concern mostly editorial matters which can be resolved through modest amounts of text editing (your examples look like that), and do not raise any substantial issues (an example of which would be where reviewers ask you to carry out additional experiments which would take a serious amount of time). In the latter case i would suggest to get in touch with the editor to discuss whether or not the additional work can reasonably be expected to be done within the time frame given for revision (if any).

I think that two things happened: the reviewers were either sloppy or they did not have the time to read the paper un-interrupted and forgot some parts of it, and secondly: you apparently made it easy enough for them to forget parts that are relevant.

Not knowing your papers, i can only give general suggestions to avoid the second issue or reduce its impact. The first is to have a proper structure in your paper: have separate (sub-)sections clarifying different aspects of your technical approach. In my sub-field of CS it is for example very common to have a (sub-)section describing the system model, which can have further sub-structure describing the fault model, load models etc -- even when I as a reviewer forgot some detail about certain assumptions or details of the experimental setup, i know where to look when such a section is there and stands out. And secondly, you can sprinkle cross-references to the relevant sections in various places ("The user did well in task X (see Section x.y for the task description) because ..") instead of repeating the task descriptions.

Fixing / re-organizing your paper along these lines should definitely be part of your reaction, don't just ignore such comments. I would recommend against bugging the editor about editorial issues -- the onus is on you to make things clear and to keep reviewers happy.

To summarise, I think your overall approach should be based on the assumption that all is your fault and not the reviewers. I am not saying it is, but i think the outcome overall is better if you treat it like that.

  • 1
    The examples for tasks were usually provided in a "Tasks" or "Materials" sub-section of the chapter "Evaluation" or "User Study". In my subfield of CS, the common structure for papers is often "Introduction - Related Work - <Novel Concept> - Implementation - Evaluation - Conclusion and Future Work" (not necessarily with exactly the same titles). Your suggestion with cross-references sounds promising. – O. R. Mapper Jul 22 '15 at 11:15
  • "forgot some parts of it" seems to be somewhat unlikely when the desired examples appear in an outstanding form such as here, namely as a separate figure. – Hagen von Eitzen Jul 22 '15 at 15:01
2

If the article is accepted and there is no rebuttal phase or step and you really had a section that called out the tasks, ignore that part of the review. If there's an opportunity to rebut, or this is a journal where the editor needs a response, then write back to the editor that you have already provided example tasks, cite the page number and section, and quote the text. There's really nothing more you need to do or should do.

2

Being a researcher myself and having seen how the review process is done, I know that there are reviewers that give the review task to their students and these students who are not experts evaluate papers written by other serious researchers. Of course, later on the original reviewer also does a check on the reviews written by the students, but I think this is a huge blow to the quality of publication. Of course, I cannot tell how much common is this practice. May be very little. And since you complained that you frequently get those negative reviews, I believe you are not a victim of these non-qualified reviewers. On the other side, as another answer mentions - a reviewer is always right. So, from your description, what I would suggest is, try to create clear description of your examples. For example, write subheadings, "Example 1", "Example 2" and this way, there is no chance even a half careful reviewer will miss it.

Remember, these days most of the reviewers are overburdened with other work, and if your paper is not very clearly readable, chances are you will get negative remarks, even when you think you don't deserve it.

-2

To be honest, I don't think a comments like

the paper does not provide any example of the tasks presented during the user study. It would greatly improve the quality of the paper if the authors could at least provide one such example

are incorrect. Although you described your surveys briefly, the reviewer may still want to actually have a look at them. If I were you, I would publish the surveys on the internet, and include a link to them in the paper. This way, anybody could easily see what your "eight tasks" are, and convince themselves that you did a great job :)

  • 3
    I do not understand. I did not just describe the surveys briefly, I showed actual questions from the survey. – O. R. Mapper Jul 22 '15 at 10:58
  • 4
    The poster wrote in his question explicitly, that there was an explicit example of a task. I don't see how insinuating the opposite is helpful in any way. – A. Donda Jul 22 '15 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.