Paper rejection based on unfair review

Question

I have seen couple of questions related to rejection of submissions to conferences (CS). Coming to my case, this is my third rejection, I agree that first two were fair decisions as the paper was not clear and needed more thorough experiments. However in this third submission I have put in all possible effort to make it the best considering feedback from prior submissions.

I received initial comments from 4 reviewers with no scores and based on their comments it was on borderline. Since there was a rebuttal phase, I have sent a response clarifying issues raised by them. I received my final notification today and it was a 'reject' decision. The final reviews had comments from an additional reviewer. His comments are atrocious and unfair. He pointed out that he is rejecting the paper as some issue X was not addressed at all. However we have addressed X with a thorough evaluation in a subsection and results put up in a graph. I'm sure that this reviewer has not read it properly. If he raised his issue may be on the way we solved X, it would have been more appropriate.

I know that appealing to the PC chairs would be a futile effort but at the same time I do not want an unfair review. This feeling of rejection based on unfair review is depressing me. How do I go about it. I feel the only way out is to resubmit it for another conference in near future.

Please advise.

Answer 1

I feel the only way out is to resubmit it for another conference in near future.

That about sums it up. Negative reviews and rejections are hard to swallow. They are common in academia and you cannot let them get to you. The best thing to do is step away from the manuscript. If you need to cry, cry. If you need to scream, scream. After a while, generally for me a week or so, try and look at the reviews again. If they still piss you off, step away from them again. Once you can read the reviews objectively, start to tackle them. This might mean agreeing with the reviewer, or seeing where the reviewer got lost, or that the reviewer is simply a fool. Once you have taken everything from the reviews that will improve the manuscript, file it away and if necessary cry and scream again. Then resubmit the manuscript to the next conference.

Answer 2

The possibility to have unfair reviews is one of the reasons why you receive multiple reviews. I have received multiple reviews where the reviewers clearly didn't review the paper carefully, and it's indeed quite frustrating.

That being said, when I looked back at the papers, I realised that they were not rejected because of one unfair review, but because I didn't manage to convince the fair reviewers: if you get four accept, then it does not matter if you have an unfair reject. Besides, it could also be the case that you've received an unfair accept review (i.e., someone who accepted the paper without really reading), so, in the end, it might balance things out.

Dealing with rejection is a huge step of the academic process, and as it has been said here in the past, if you never get papers rejected, you are probably not aiming high enough. As long as you have only one unfair review, ignore it, and move on to the next conference (and if you only receive unfair reviews, well, you are probably submitting at the wrong venue).

Answer 3

As you said, the paper was on the borderline after 4 (fair) reviews. I would advise first to discuss the paper and the results with your supervisor / colleagues, and if you are still determined to continue, rework it significantly, taking in account the criticism of four referees, that you consider as fair. Make sure the result moves far from the borderline (and in the right direction). Then re-think the remarks of the last ("unfair") referee and decide what you want to do with them. Probably, you should not trash them away completely — try to find a motivated comments and suggestions apart from the "unfair" ones.

When a new version of a paper is ready, you may wish to show it to your supervisor and send to several colleagues outside your group asking them for some pre-submission comments — this may save you a lot of time.

Answer 4

Are we talking about the same paper that is rejected a total 3 times? If so, you should seriously consider if this paper (however you improve it) will ever be good enough. Perhaps the problem you solve is not interesting enough or your solution does not promote the state-of-the-art significantly enough. In that case, maybe aiming too high is your problem. Also, note that in prestigious CS conferences (I assume this is your case, due to a rebuttal phase), borderline papers never make it. At the end, there are enough papers who pass with flying colors and the rest fail.

You should start considering submitting to a lower ranked CS conference or perhaps a workshop. You have already wasted a full year or more (that is what probably the time required for getting those 3 rejections)and wasting another 3-6 months for the next conference, might not be feasible. You should seriously start discussing this possibility with your supervisor.

I understand that rejection hurts but many of us had works that were rejected, kept improving it and got rejected again. At some time, it is time to cut your losses and move-on. I think it was @JeffE that said "The universe has spoken. Move on" and he is right. Submit to a lower conference / workshop to at-least patent those results and lower your sense of rejection, but without wasting too much time further improving those results. Instead focus on more interesting variations and move-on. But most-of-all discuss all these possibilities with your advisor

Answer 5

In Hebrew there's a saying:

צרת רבים - חצי נחמה

which means

The trouble of many is half a consolation

Unfair reviews seem to be very common. I just got a set of reviews a few months ago with one reviewer saying (roughly) "section X was great but section Y was weak" and another reviewer saying "I really liked section Y; unfortunately, section X was sub-par" and they made conflicting suggestions regarding what to expand on and what to mention in passing. And then there are people who seem to just not be close enough to your field of research who review the paper in a way that makes that fact abundantly clear. etc.

So while this answer isn't any sort of practical advice, at least know that you're not alone with this experience.

Answer 6

I think that the real problem that may however be hard to address, is that there is a schedule to visit certain conferences well in advance and that you are supposed to come up with research results that will be good enough for those conferences. This is not what I am used to doing, but then my field is not CS. If you could simply work on your research program until you get outstanding results, then getting those results published in a decent journal won't usually be a problem. There will still be cases of bad refereeing but then you won't mind if the referee wants things that are already explained to be clarified if the report starts with the statement that the paper is excellent and should be published.