35

I am reviewing a paper of a top-tier computer science conference. After reading the abstract and introduction of the paper, I think that the paper does not have that standard to make it up to the conference reputation. In other words, the problem that they have solved is a minor one and perhaps belongs to some low-tier conference.

Can I simply recommend to reject the paper without reading it further? Is it right to do so? I am feeling that it is a waste of time to read it further. Or should I must read the full paper and verify their claims and suggest the improvements (if any)?


Note: I am somewhat new to reviewing; this is the 6th paper that I am reviewing this year.

5
  • 19
    Would it take you more than a few minutes to give a light pass over the remainder of the paper?
    – Nat
    Sep 28 at 6:44
  • 4
    Please don’t write answers in comments. It bypasses our quality measures by not having voting (both up and down) available on comments, as well as having other problems detailed on meta. Comments are for clarifying and improving the question; please don’t use them for other purposes.
    – cag51
    Sep 29 at 17:11
  • 1
    Trivial case: abstract is gibberish. Should question be better?
    – Joshua
    Sep 29 at 21:05
  • 1
    @Nat, what would "a light pass" done in a few minutes accomplish? That is still not reading the entire paper.
    – Tripartio
    Sep 30 at 6:09
  • 1
    @Tripartio I have seen papers where the abstract and intro manage to "bury the lede". (That's the fault of the authors.) I have also read abstract/intros and totally misunderstood the point of the paper until I read on, then go back and realize the point was right there in the abstract but I just failed to grasp it. (That's my fault.) I would argue that a good reviewer should avoid both traps. A light pass will generally be sufficient to at least see if more attention is merited.
    – Mike
    Sep 30 at 15:51

13 Answers 13

62

It is of course possible that a single portion of a paper will be sufficient for rejection, so you can recommend rejection on the basis of a single part alone. However, you should remember that the best practice for a peer-reviewer is not just to reach a recommendation but to offer useful advice to the author on how they might improve the paper. As a reviewer, you have an opportunity to help the authors of a paper make it better for the next submission.

As a secondary matter, except in extreme cases, failing to read the entire paper may be viewed as a bit of a dereliction of duty by some people --- I would certainly view it that way. Even desk-rejections will almost always say that the editor read your paper, so even here, the author is given the courtesy of having their paper read in full.

12
  • 27
    As a secondary matter, failing to read the entire paper may be viewed as a bit of a dereliction of duty by some people - Maybe by some people, but in my field (pure math, where papers can be quite long and take time to go through the details), if it is a clear reject I would regard reading the whole paper as a waste of reviewing effort.
    – Kimball
    Sep 28 at 11:07
  • 3
    @Kimball I remain unconvinced that in your discipline it's any different than in others. If it's a complex and long paper the attention to detail will be different, but the idea stands. Sep 28 at 11:19
  • 14
    +1 for the sentence "However, you should remember that the best practice for a peer-reviewer is not just to reach a recommendation but to offer useful advice to the author on how they might improve the paper."
    – MAS
    Sep 28 at 11:20
  • 12
    @MrVocabulary This could be a debate about semantics, but it somebody spends 100 pages proving to their own satisfaction than 1+1=3, and the introduction and abstract state that 1+1=3 is a key result in their work, I'm unlikely to want to read the details except for personal amusement.
    – alephzero
    Sep 28 at 14:01
  • 11
    @alephzero sure, but is that likely and frequent enough that we should focus on that when discussing general rules and best practices? Sep 28 at 17:48
28

must read the full paper and verify their claims and suggest the improvements

Read again the letter editor sent you. Why did they want your input on the paper?

In my field, the review is not about "rejecting" or "accepting" but about finding errors and flaws, as well as good points, and assessing the value of the artefact. The editor will decide whether these issues are suitable for rejection, acceptance, or acceptance with modifications.

For example, here are the top-level questions that editors ask reviewers in PNAS:

  • Suitable Quality?
  • Sufficient General Interest?
  • Conclusions Justified?
  • Clearly Written?
  • Procedures Described?
  • Supplemental Material Warranted?

As you can see, only one question is about suitability for the outlet. The others are about the quality of the research paper as it stands.

1
  • 3
    A CS conference paper review is unlikely to come with a letter from an editor (and it may not be any individual's decision). Sep 28 at 3:41
22

I have two points, one objective and one subjective, why you should not do this.

Objectively, it is quite common to form an opinion of a paper only by reading the abstract and introduction. I have heard that most reviewers decide by the end of reading the introduction whether they want to recommend revision or rejection; reading the rest of the article does not usually change this initial impression significantly. However, sometimes a promising introduction might fail to deliver and so an initial inclination towards recommending revision might change to a recommendation of rejection. On the other side, sometimes a lackluster introduction might be followed by an article with real potential, in which case the reviewer who initially felt inclined to recommend rejection might realize that the introduction needs to be rewritten to properly highlight the value of the article. So, objectively, you cannot properly evaluate an article until you have completely read it. Bad beginnings might end well, just as good beginnings might end badly.

Subjectively, it is irresponsible and even morally wrong to claim to give a scholarly review of an article that you have not read completely. Would you write in your evaluation to the article, "My recommendation for rejection is based only on reading the abstract and introduction of the article; I saw no need to read further"? If you would not openly write that, then that should tell you that there is something wrong with doing that. I am sure that you would feel rather upset if you found out that a reviewer of one of your articles did that to you. So, subjectively, it is wrong to recommend rejection of an article you are asked to review if you have not read the entire article.

2
  • 1
    Should "end of reading the instruction" be "end of reading the introduction" (crosses fingers and hopes that you meant introduction!). Also, as an author you can have "good" reviews and still get a rejection.
    – Pam
    Sep 29 at 14:13
  • 1
    @Pam, Yes; thanks for pointing it out. I have corrected the typo.
    – Tripartio
    Sep 29 at 14:23
18

As a small addition to the other answers:

Whatever you choose to do, please be honest and open about it towards the editor, the other reviewers and the authors. If you really made your recommendations after reading the abstract and the introduction, (and think that was quite enough), then say so in your review. This helps the others in putting your review in perspective.

17

Look at this through the author's perspective. They spent time working on this research and writing the paper - probably because they believe it is important enough to be worth their time and the readers' time. Now, as a reviewer, you may think that this is wrong and the research is not so good to merit publication, because

the problem that they have solved is a minor one and perhaps belongs to some low-tier conference.

Perhaps you are right and the author's are wrong. But how helpful would it be for the authors to receive a rejection stating only this? Surely, they would like to know how to become a better researcher and how to increase the chances for their next paper to be accepted. Can you help them with it? Can you offer constructive critique and helpful suggestions for improvement? This will make the whole experience more pleasant and rewarding and ultimately worth everyone's time.

12

As others have pointed out, your responsibility is to help the editor decide what to do.

I don't think you can fairly recommend rejection without actually reading the paper. You could write the editor declining to review because, based on the abstract and introduction you don't think the paper is significant enough to warrant the time it would take you to read the manuscript carefully.

Then the editor can find other reviewers, or agree with your assessment and desk reject.

10

In theoretical CS I wouldn't recommend this. If mathematics I would definitely suggest it would be improper. You don't have enough information on what the future value of the paper might be.

It is possible, though unlikely, perhaps, that the techniques used to prove a "minor result" are actually more valuable than the result itself. Those techniques might, in theory, be used to establish other, much more important, things. How something is done can be much more important than what is done; certainly in mathematics and possibly in other theoretical fields.

I'll agree that the authors should have pointed this out, of course, if they thought it was important.

If you told me, as an editor or conference program chair, that you'd done this, I'd likely feel like I needed additional advice before rejecting the paper. You are admitting to providing little advice of value.

8
  • 8
    I'd disagree with this specifically for mathematics, though that is in the context of journals, and because reading the paper carefully may take another several weeks if not longer. If a paper is not suitable for a journal, you should recommend rejection QUICKLY so that the authors can quickly submit it somewhere else. Sep 27 at 19:13
  • 3
    @AlexanderWoo, actually, for mathematics you would be wrong. Sometimes the proof of a theorem is far more important than its statement and gives far more insight into how things fit together. I'd expect a mathematician to know that. In particular, you don't gain insight into mathematics by reading a bunch of theorem statements.
    – Buffy
    Sep 27 at 19:27
  • 9
    @Buffy: No, if they hadn’t mentioned it in their introduction, I wouldn’t feel foolish at all! More seriously, you are of course entitled to your opinion, but it is not an accurate statement about the practices of mathematics journals, and I worry that you will mislead any early career mathematicians who happen to read this thread. The fact is that maths journals often reject papers based on the importance of their results rather than correctness, and that importance is usually primarily judged on the results stated in the abstract and introduction.
    – HJRW
    Sep 27 at 22:39
  • 4
    I reckon a professional non-isolated mathematician would reasonably estimate the contents of their own paper. If the proof is what's important/novel, they'd state that in the abstract/title, wouldn't they?
    – Pablo H
    Sep 28 at 15:49
  • 2
    Just to add to this... admittedly I'm a former theoretical physicist, not a mathematician, but I would think if Paper X proved some obscure result, but used a technique that could be applied to prove the Riemann Hypothesis, and didn't point that out, then a perfectly valid novel Paper Y would be to take the technique and actually solve the Riemann Hypothesis (with a proper citation to Paper X). In my opinion, it shouldn't be the referee's job to realize that Paper Y could be written, if the author of Paper X didn't point it out in the abstract/introduction/discussion.
    – Andrew
    Sep 29 at 17:19
10

There are two scenarios both consistent with the data that you have described.

  1. The paper makes only a minor contribution to the field and you should recommend rejection from this journal.
  2. The authors have found an important result but failed to effectively describe/"sell" its importance in the abstract and introduction. This is not uncommon when researchers are wrapped up in their own field, they see the importance and feel it should be obvious to the reader. Good scientists are not always good writers. Here you might need to recommend revisions in order to better highlight the contribution of the paper.

If you stop reading after the introduction, you don't know which scenario is the case, and you are doing the authors a disservice by assuming (1).

If you were submitting work for feedback, would you feel satisfied if someone read only the first section and returned it to you saying it wasn't good enough?

5

If a math journal is asking for a quick opinion, then yes it's ok to recommend rejection based on just the introduction but you must actually read the introduction carefully! If the introduction clearly lays out the main results of the paper and you think those results aren't interesting enough to merit acceptance at the level of the journal where it's submitted then there's no need to read the rest of the paper carefully.

By contrast, if they've already gotten quick opinions and are asking for a more thorough refereeing job then I think it's inappropriate to not read the paper more thoroughly. Remember that other people may have different opinions about the merits of the paper.

2

Based on the other answers, I guess this will probably be a contrarian opinion, although to be honest I suspect this is probably how many people behave even if they don't admit it. I don't think you have an obligation to thoroughly read the paper, although I think you should spend, say, an hour or so going through it. For a top tier journal / conference, "results are not sufficiently impactful for this venue" is a valid reason to reject the paper. I think you should read the discussion, main figures/results, and read through the text (not necessarily checking every step) to make sure that you understand what is there and can summarize it. You should at least be able to confirm that the content of the paper matches what is in the abstract and introduction, or else find if there is something in the paper that you need to look into in more detail because it could change your opinion.

In an ideal world, every referee would thoroughly read every paper they were sent and send detailed feedback on how to improve the paper. In the real world, people are busy, and if you already think the headline results of the paper are not sufficiently interesting for the top tier journal/conference you are reviewing for, I don't think you are obligated to spend days checking every claim.

It would be different if the journal/conference were mid or lower tier... then I would tend to say that something correct and novel could be publishable even if the impact of the main results was not very large. Judging correctness requires more work to actually read the paper and check things, compared to judging the importance of the headline results, assuming you have a good sense of the quality standards in the field.

1
  • I feel this is the only answer that really considers what OP asks: "must I read the full paper and verify their claims", to which I agree that the answers are "yes" and "no" respectively. Sep 30 at 8:23
1

I’ll sacrifice politeness for the sake of clarity: If you have time to write here - you have time to read the paper. You may feel that you can’t understand it (I’ve seen such badly written) but at least you should give a try.

1
  • 4
    Reading a paper is not the same as reading a comic book. You need to think alongside what is written. You need to develop an intuition of their idea and much more.
    – IY2
    Sep 28 at 15:21
1

(copy-pasting directly from the OP comments, to enable room for discussions)

In some (most?) top-tier CS conferences, papers are screened before they are assigned to reviewers. Usually, this involves checking for obvious plagiarism, but it can also extend to verifying whether the manuscript is (just from its looks) acceptable. The idea is to maximize the gain from reviewers: assigning 5 papers that do not even "need" reviewing to be rejected would waste the reviewer's time; and would also waste the reviewer potential contribution to the proceedings.

If the paper passed such screening it's likely that the abstract was deemed "somewhat" appropriate for the venue.

You should read it thoroughly and - moreover - question whether YOU are the problem (ie, lack of expertise, which is common and acceptable): if this is the case, then avoid clear statements and do mention that your recommendation is just an educated guess.

If the abstract is really, really bad, then you can still recommend a rejection. But - for the sake of science - try to provide at least some additional motivations.

0

I wouldn't base a decision to accept or reject a peer-reviewed paper based solely on the abstract and introduction sections and an opinion "the problem that they have solved is a minor one." Perhaps there is more in the paper that authors haven't explored and perhaps their focus is not correct.

There's a plugin in chrome called Read Shit Faster (ugly name but what a powerful tool it is) that allows you to read the paper in a super-easy manner allowing you to detect major flaws almost instantly and know what to look for in a more thorough reading.

The order in which I read carefully these contributions goes like this:

  1. Abstract: What am I suppose to learn from this piece? What is it that the authors claim to have found that has not been addressed before?
  2. Conclusions: Yes, I go to that section right away to see if it is consistent with the abstract and if they are delivering what was promised.
  3. Methods: If this section has problems from the experimental plan, setup, tools, etc., I will start considering rejection.

By now, you should have seen if the language is correct or if there are major problems in this department. I collect those right away to either help the authors write a better paper or simply to justify a rejection in case there is no substantial contribution.

If at this time I consider that the authors have something worth reading, I'll read their Results and Discussion section to check for consistency with the sections that I have already read and then I will move to he references section to see how up to date is the bibliography. By the way, I would dig some extra references that are more current to suggest some improvements for the authors and perhaps to see if there's something remotely similar that has been published... Sometimes, if you know the authors, you may see if the work has been published before and point those works to the editor. Turn it in can also detect plagiarism (intentional or not) and you may point that to the conference organizers.

Typically, by the end of a Read Shit Faster session I would have made my mind in regards to reject, major editing needed, or accept (rarely) a paper. The remaining process is to document everything.

Not the answer you're looking for? Browse other questions tagged or ask your own question.