How to React to Incorrect Claims by Reviewers?

Question

As this has happened to me several times for different papers now, I am not describing a specific case here, but the abstract aspects that were common to all situations.

Occasionally, in the "Tasks" or "Materials" section of the "Evaluation" or "User Study" chapter, papers of mine contain statements such as:

• "Figure ... shows one of the tasks from the user study. The correct answer was ... ."
• "For the user study, we had prepared eight tasks. They were given in the form of natural-language questions. Users were supposed to use our novel concept to input the appropriate parameters as described by the questions. Questions ranged from simple (such as '...?', see Figure ... for the solution) to complex (such as '...?', cf. solution in Figure ...)."

In these cases, the figure captions once again mention the figures depict questions and/or solutions from the user study.

Occasionally, these are met with reviewer comments such as:

• "The paper does not provide any example of the tasks presented during the user study. It would greatly improve the quality of the paper if the authors could at least provide one such example. Without this, I find it very hard to get a clear idea of how the study was conducted, and thereby, I also cannot tell how reliable the results presented in Section ... are."

I base my reasoning on the idea that this reviewer comment is, plain and simple, incorrect, as I did provide concrete examples of tasks. If I am somehow mistaken about this, please do let me know.

While such statements were certainly not the only reason for voting to reject, some of the reviewers who wrote something along these lines made it rather clear that they saw the alleged lack of any task examples as one of the most critical issues with the submission.

My question focuses on the situation that such a paper is getting revised - either, because it was accepted after all, and I am preparing a camera-ready version, or because it was rejected, and I am trying to improve it before submitting it elsewhere: What is the appropriate reaction with respect to such comments?

• Ignore the statements and do not change anything about the respective examples in my paper. This might be a viable way to go, but it leaves a bad taste to completely ignore some reviewer statements, even though they are clearly comprehensible and should be straightforward to implement.
• Re-word the sentences mentioning the examples. I might do that, but I am wary of making things worse, as I already consider the respective statements in my paper quite unambiguous as they are.
• Add more examples, and possibly at different locations. This could make it harder to miss the examples, but in CS conference papers, space is an extremely scarce resource.
• Ask the editor. Especially in cases where the paper was eventually still accepted, this seems quite extreme to me. I feel contacting the editor should be reserved to truly exceptional and severe issues, not "every-day worries" of all authors who are preparing their CR-versions.

Note that I am not asking how to deal with the review decision. Even though it is slightly annoying if a rejection is indeed based on incorrect reviewer statements, I am ok with resubmitting a revised version elsewhere. I am specifically asking how to revise the particular parts that were incorrectly criticized.

Some peculiarities of my field to note that might not be obvious:

• Only some conferences have a rebuttal phase. If they do, it is not guaranteed (or even reasonably likely) that especially reviewers who criticized a paper a lot will actually read the rebuttal and possibly even amend their review. Also, these rebuttals are usually limited in length (e.g. to 500 words), so you can usually mention only a selection of all issues raised by reviewers.
• CR-versions are rarely accompanied with explanatory letters to the editors (or, more precisely, the program committee) in my experience.
  • Submission usually happens by means of web-based submission systems which allow the upload of exactly one PDF file, which is sometimes automatically checked on formal factors, such as number of pages (so even tacking an extra page with explanatory notes in front of the paper will not work).
  • If an opportunity is provided to upload some additional files, this is usually meant for uploading sources and supplementary material such as videos that will be put on the proceedings CD. It is not clear whether the material will be sifted thoroughly enough so letters to the editors will be found in time and treated appropriately.
• My field is related to HCI.
  • Therefore, contributions are sometimes graphical in nature. Consequently, figures sometimes do not just provide information about the contribution (statistical graphs, exemplary architectures), figures are often a crucial part for showing the contribution. In other words, sometimes, the depiction is the contribution, or it shows how the novel concept presented in the paper does something.
  • User studies are used to test how well human users can get along with the novel concept. Therefore, tasks are often written as questions. Study participants then have to find the correct answers to these questions by using the novel concept being tested. This usually means that they have to interact/make inputs with/in the implemented prototype that shows the novel concept. This is meant to mimick how the concept would be used in real life, where users would use the respective novel HCI technique for retrieving information, as well.

Answer 1

I almost never ignore a reviewer comment. Even if the reviewer is simply wrong, the way in which they are wrong usually points out a way that some significant subpopulation of readers is likely to misread the paper.

Given this, when a reviewer seems to have simply overlooked a chunk of the paper, I will generally fiddle with the wording, the structure or the introductory material in order to make it pop out more clearly.

For example, with your "eight tasks" example, I might do any or all of the following to make the presence and visibility of the material more blatant:

• Make sure that the introduction has a "the structure of this paper" subsection that includes the word "example" in it.
• Do the same for the section including the tasks with a "structure of this section" paragraph at its beginning.
• Add a subsection header for "Examples of Tasks"
• Put the question examples into a bullet list, rather than prose

Remember, the reviewer may be incompetent and sloppy, but many of your readers will be also. You want to get your point across to as many of them as possible, regardless.

Answer 2

A simple rule of thumb can help to handle reviewer comments: "The reviewers are always right, even when they are wrong."

This means that when reviewers make wrong assumptions or draw wrong conclusions, apparently the article is not clear enough and allowed them to do so.

Mostly, in my experience, the solution that works best, is to restructure the article to prevent readers from drawing wrong conclusions. Make the text clearer, improve the structure, avoid being overly verbose.

The reviewers will appreciate such an approach more than you trying to point out in the reply-to-the-reviewers why they are wrong, or just ignoring their comments.

When your article was rejected immediately, then use this approach to improve the paper for a new submission, and prevent running into the same issues.

Answer 3

The following is predicated on the assumption that the reviewer comments indeed concern mostly editorial matters which can be resolved through modest amounts of text editing (your examples look like that), and do not raise any substantial issues (an example of which would be where reviewers ask you to carry out additional experiments which would take a serious amount of time). In the latter case i would suggest to get in touch with the editor to discuss whether or not the additional work can reasonably be expected to be done within the time frame given for revision (if any).

I think that two things happened: the reviewers were either sloppy or they did not have the time to read the paper un-interrupted and forgot some parts of it, and secondly: you apparently made it easy enough for them to forget parts that are relevant.

Not knowing your papers, i can only give general suggestions to avoid the second issue or reduce its impact. The first is to have a proper structure in your paper: have separate (sub-)sections clarifying different aspects of your technical approach. In my sub-field of CS it is for example very common to have a (sub-)section describing the system model, which can have further sub-structure describing the fault model, load models etc -- even when I as a reviewer forgot some detail about certain assumptions or details of the experimental setup, i know where to look when such a section is there and stands out. And secondly, you can sprinkle cross-references to the relevant sections in various places ("The user did well in task X (see Section x.y for the task description) because ..") instead of repeating the task descriptions.

Fixing / re-organizing your paper along these lines should definitely be part of your reaction, don't just ignore such comments. I would recommend against bugging the editor about editorial issues -- the onus is on you to make things clear and to keep reviewers happy.

To summarise, I think your overall approach should be based on the assumption that all is your fault and not the reviewers. I am not saying it is, but i think the outcome overall is better if you treat it like that.

Answer 4

Being a researcher myself and having seen how the review process is done, I know that there are reviewers that give the review task to their students and these students who are not experts evaluate papers written by other serious researchers. Of course, later on the original reviewer also does a check on the reviews written by the students, but I think this is a huge blow to the quality of publication. Of course, I cannot tell how much common is this practice. May be very little. And since you complained that you frequently get those negative reviews, I believe you are not a victim of these non-qualified reviewers. On the other side, as another answer mentions - a reviewer is always right. So, from your description, what I would suggest is, try to create clear description of your examples. For example, write subheadings, "Example 1", "Example 2" and this way, there is no chance even a half careful reviewer will miss it.

Remember, these days most of the reviewers are overburdened with other work, and if your paper is not very clearly readable, chances are you will get negative remarks, even when you think you don't deserve it.

Answer 5

If the article is accepted and there is no rebuttal phase or step and you really had a section that called out the tasks, ignore that part of the review. If there's an opportunity to rebut, or this is a journal where the editor needs a response, then write back to the editor that you have already provided example tasks, cite the page number and section, and quote the text. There's really nothing more you need to do or should do.

Answer 6

To be honest, I don't think a comments like

the paper does not provide any example of the tasks presented during the user study. It would greatly improve the quality of the paper if the authors could at least provide one such example

are incorrect. Although you described your surveys briefly, the reviewer may still want to actually have a look at them. If I were you, I would publish the surveys on the internet, and include a link to them in the paper. This way, anybody could easily see what your "eight tasks" are, and convince themselves that you did a great job :)