5

Many Computer Science conferences offer a rebuttal phase, and it often does not require the authors to upload an updated version. Instead, the authors clarify things and many also make promises. What are reviewers supposed to do with these promises? Should the reviewer trust it or ignore it? The promises may not be fulfilled at all eventually.

1

I thought about this a lot when I first started reviewing for conferences and discussed this with other people. I will give a sum up how I handle it as a result of these discussions but it seems to be a quite controversial question.

I choose to give the benefit of the doubt except there are explicit reasons not to. Rebuttal periods are normally one week and limited to 1 or 2 pages, there is only so much additional results that can be produced and included. Also, generally doubting the sincerity of all authors is not feasible on the long term. You might also doubt that the numbers they report are not true because you do not have the code to check it.

Although it was not really the question, the second part is about how to avoid only getting promises:

  1. Precisly state in the review which of your concerns are crucial for acceptance. I often see reviewers who do not make it clear what their priority for the rebuttal is and then the rebuttal is full of replies to minor comments. (For example, stating all the parameters used for training is useful in the paper but not useful in the rebuttal.)

  2. Phrase your actual concerns instead of saying which additional experiments/comparisons would ease them. (The difference between "Please compare to initializing with this baseline" (easily promised) and "I do not see why this complicated strategy should be better than this simple baseline") In my experience, this leads to more interesting answers.

| improve this answer | |
1

Most conferences will have the PC chairs provide guidance, and ultimately make decisions, in these matters. So it's not strictly speaking something that a reviewer or regular PC member needs to worry too much about.

However, from what I have seen in Software Engineering conferences, "promises" are largely ignored in rebuttals. Rebuttals are seen as a way clarify things that the reviewers may have misunderstood or missed, not as a "response letter" where the authors vaguely promise changes in the camera-ready version. Some conferences have even made this explicit in the call - they specifically limit the rebuttal to reactions to reviewer comments and clarifications of material and data that is already reported in the original submission. That is, the following rebuttal is ok:

Reviewer 2 seems to have misunderstood the scope of the submission and think that we have only looked at narrow scope A, while we have actually researched the wider scope A+B. We agree that this was not sufficiently clear in the description of the research method, but we believe that this can be fixed with an improved textual description of the method in Section 2.

Conversely, the following rebuttal will probably not get you far:

Reviewer 2 was disappointed that the scope of our study was only narrow scope A. However, we have additional data related to B and will present these results in the camera-ready version.

(note that strictly speaking the first version also contains a small promise, but it's a simple one of a small textual change - most committees will give the authors the benefit of doubt here, and even if the authors don't make good on this promise the world will probably not end)

That said, many conferences now seem to implement a "shepherding" mechanism, whereas (some) authors only receive a conditional accept under the condition of implementing what they promise in the rebuttal. In that case, either the PC chairs or a designated PC member will follow-up after submission of the camera-ready version whether the main comments have been addressed as promised. This is not typically done for all, or even for most, papers, but only for a small subset of papers where the reviewers asked for rather specific changes, which are both essential enough that the paper can only be accepted after they have been implemented and small enough that they can realistically be done in a day or two. One example that I have recently seen was a highly relevant and interesting study, where the reviewers felt like the authors were overselling their results in one specific aspect. The reviewers wanted to make sure that this was removed in the camera-ready version, and this was assured through a conditional accept and shepherding.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.