26

I am asked to review a paper for a fairly highly regarded conference.

The paper presents some 'novel' algorithms to do a particular job, but it fails to mention any of the various related previous work which address the very same issue.

Even if the algorithm is sound and their method is thorough I am hesitant to accept the paper, since there is no comparison to existing techniques; neither on a methodological level nor on the level of results.

Can I reject such a paper based on the non-existent comparison to related work, or should I focus more on their method and results itself?

  • 3
    Does it work, or does it not? And, are you competent to judge it? What is actually the question here? Is whether you should spend your tie on it? I can't see any other reason to ask us. – Mawg says reinstate Monica Sep 5 at 8:43
  • Would people attending the conference be reasonably expected to already know the related work, if so it may not be a good use of their time to cover it. – Ian Sep 5 at 9:32
  • 5
    Just a minor point: the decision is not yours to reject or not. As a reviewer, you can only recommend. – TonyK Sep 5 at 18:28
  • Can you please explain what "paper for a conference" means here? There are a lot of conferences in a lot of different subjects and none of them is run in the same way. Talk? Poster? Conference proceedings? One, four, twelve pages? Paper is a useless slang term, correctly understood only in your little neck of the scientific woods. – Karl Sep 6 at 18:27
  • This depends whether method is a replacement or complementary. An improved compression algorithm needs comparison to others - it is one or the other. Well, unless it is radically different and interesting because of that. A physical simulation algorithm is likely an alternative to the current ones and will of course need many comparisons later on, but there is no need to have any in the first paper beyond mentioning those other methods exist. – Zizy Archer Sep 7 at 6:38
53

One of the key questions for any piece of scientific work is this: how does this work contribute to human knowledge?

If a work fails to even discuss its relationship to prior work, then it is entirely appropriate to reject it. Likewise, if the authors mention algorithms that are directly comparable but fail to actually make a comparison with any of those algorithms. When I am given a paper that is entirely isolated in this way, I will not even bother to check the details of the algorithm's soundness, since the work already has a major disqualifying failure.

For many algorithms, however, the differences are qualitative and not quantitative, and thus a direct comparison would be unenlightening or pointless. For example, if algorithm X tolerates a class of failures that prior algorithms do not, then it is enough to show that algorithm X tolerates those failures. One does not necessarily need to re-prove or empirically demonstrate that the others do not.

Likewise, the number of potentially related prior works is often vast and complicated. No algorithm paper will ever compare to all of the other related algorithms, because "related" is a broad and fuzzy concept. You and the authors will generally not make the same judgements about which algorithms might be interesting to discuss in related work or to directly compare against. Thus, the standard to which you should hold related work is not "Do I want to see comparison against Algorithm Y?" (that's Evil Reviewer #2 behavior) but rather "Does the comparison provided sufficiently support the author's assertions?"

Bottom line: be generous in your judgement of comparison to prior work, but papers with no meaningful comparison should be rejected.

| improve this answer | |
29

Needs revision

There is a meaningful difference between asserting that a particular work is fundamendally not appropriate for this venue, or that it is unacceptable as it now stands. The former is a 'reject', the latter is 'accept with revisions required'.

You should recommend a rejection if the changes required to make it a good, appropriate paper would require to fundamentally change the paper itself or its conclusions. According to your description it's not necessarily the case - you claim that the paper needs a comparison to existing techniques on both a methodological level and also on the level of results. If the paper added these comparisons, would the paper be acceptable, or would it still be not good enough? That's what the review should state.

| improve this answer | |
  • 1
    From a conference paper I would expect at the minimum a comparison at a methodological level with existing techniques, at the bare minimum stating why they are not a good way to solve the particular problem at hand. – EarlGrey Sep 7 at 13:28
21

You can write: the algorithm is sound and their method is thorough, however, there's no mention of the various related previous works (e.g., X, Y, and Z) which address the very same issue, without such discussion I cannot evaluate the work's novelty and I must reject the paper at this time.

| improve this answer | |
9

I will add one important point not addressed in other answers so far: I consider that rejection criteria for a journal to be different from that for a conference, even for the same article. The most pertinent point I see here is that whereas journal articles have the time to request multiple rounds of revision (which could drag on to a year or two or even longer), conference reviewers must decide immediately whether the article can be quickly made acceptable, or not, since the conference review and presentation timeline is always very tight. In journal submission terms, a conference submission can give only one of two decisions: either it is "accept with minor revisions", or it is "reject". In particular, there is no option for "major revisions" as a decision.

With this understanding, the question is: whatever the revisions you think might be necessary, can they be done easily and feasibly in the space of just a couple of weeks? If so, you can accept the article and request those minor revisions. But if the revisions you consider essential cannot be done easily or quickly (in the timeframe of the conference, specifically, before the deadline for authors to submit the final "camera-ready" proofs), then you should recommend rejection.

Note that my answer is very general and does not even address your substantial question of whether or not the absence of a literature review is sufficient to reject the article. Rather, I ask you to reframe the question thus: do you have confidence that, in the tight timeframe of the conference, the authors can fix the literature review problems? Also, note that, unlike with a journal submission, you probably will not be given the opportunity to verify if they have done what you consider essential.

With this understanding, based on the details in your question, I think it might be unlikely that the authors would satisfactorily reframe the article in the literature, especially since you would not be able to verify this. If so, you should probably recommend a rejection. But, of course, explain to the authors what you would have like to see, so that they can hopefully do the revisions and submit to another conference or to a journal.

| improve this answer | |
  • Conference papers are often submitted after the conference as well, and sometimes there are 3-6 months available for submission. Similarly, there may be ample time for review. I have been asked to review a conference paper where suggesting revisions was an option. – Jake Sep 6 at 9:29
1

should I focus more on their method and results itself?

If you do this properly then you will have to do the comparison.

It should be the responsibility of the author to compare and contrast.

Of course if the new algorithms solve a previously unsolved problem this may be sufficient.

| improve this answer | |
0

If it were me, I would say that it needs a substantial revision before being considered again for submission and that they need to add a thorough lit review making it clear that they are aware of the previous work on this problem and can explain the novelty of their contribution.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.