What to consider as reviewer before dooming a paper

Question

I am conflicted about what should influence my recommendation as reviewer.

A few times, I've been in a situation similar to the current one. Consider a paper that, after receiving a major revision, has been resubmitted, and I am doing a second-round review. Or alternatively, think of a paper currently in the rebuttal phase of a peer-reviewed conference. After the initial round of reviews the situation is the following:

• 2-3 reviewers are somewhat positive about the paper, but not overwhelmingly positive. Their reviews have reasonable quality, but do not go into depth.
• (At most) one reviewer is negative about the paper and highlights one or more considerable but not necessarily critical weaknesses (e.g. correct but outdated methodology, imperfect reproducibility). The review is quite detailed and it is clear they know what they are talking about.
• I am reserved about the overall idea and have identified some shortcomings that could be improved. The revision provided by the authors addresses some of the shortcomings, but not all. I do not feel that the remaining issues are huge problems, but neither are they negligible. Quality-wise, I believe the paper could be accepted, but I would prefer a rejection.
• Experience tells me it is likely that if I raise my rating/recommendation, the paper will be accepted, but if I stick with my rating, then the paper is doomed ( it's a highly competitive venue).

No further round of reviewing is permitted, and the next decision is final.

As stated, I am reserved about the paper's idea and would prefer if the paper is rejected. However, this is science, and it should not be about my "feelings". The paper does not have a major technical flaw and fits to the venue. It could be accepted. But then again, many other reviewers and editors sometimes follow their feeling of gut. Peer-review is inherently subjective. Maybe I should just follow my feelings as well. However, I can also empathize with the author side: More than once have I or my colleagues been in a similar situation. This thought makes me want to help the authors and raise my rating. But again, this is science, and such factors should not matter – at least I think so.

You will probably realize that I am conflicted.

Question: Which of the above factors should influence my decision? Are there perhaps other, more important factors that I should consider?

---

I am aware that the final decision is made by the editor/program chair, and not by me. However, I am still concern about the impact my recommendation might have.

Answer 1

You should consider whether you think the paper itself should be rejected.

If that is from a gut feeling then applying a lower confidence to your review seems appropriate. But tactical considerations on how others rated the paper, or how your rating relates to others, or what the authors deserve, or similar are not the point of a scientific review.

Answer 2

Some random points to consider:

• Does this science deserve to be published, and is this venue appropriate? By "deserve" I mean was the study appropriately conducted. Are the results meaningful to the community?
• Not every paper has to conclude every imaginable question one could have on that topic. Reviewers often lose focus and expect a career's worth of work out of every manuscript. If it answers some questions, and opens up new ones that inspire further research, then that is great.
• Do not fall into the role of gatekeeper but stay as a peer reviewer. You know the difficulties involved with research and that not everything is perfect. Be a peer, approach it in good faith, and then decide if the work lives up to the claims and the venue. If you see other reviews, were they done in good faith?
• All things being equal, if there are no major technical flaws or misrepresentations, does the field benefit more from this article getting published in that venue with its respective visibility, or if this work ends up in a lower tier (but maybe more specialized) journal?

Answer 3

The only way to leave your feeling out of things is to stick to facts, and facts alone.

Firstly, where at all possible, a refrain from giving ratings during per review. To the extent I've stopped reviewing for some venues that won't let you submit a review without giving them.

What I would do in your situation is write a review that said:

I am reserved about the overall idea and have identified some shortcomings that could be improved. The revision provided by the authors addresses some of the shortcomings, but not all. I do not feel that the remaining issues are huge problems, but neither are they negligible.

In addition to this I would evidence my points. In particular I would highlight the remaining non-negotiable remaining problems. I would not make a recommendation either way for accept or reject.

This is what @R1NaNo means by being a reviewer rather than a gatekeeper.

Answer 4

Experience tells me it is likely that if I raise my rating/recommendation, the paper will be accepted, but if I stick with my rating, then the paper is doomed ( it's a highly competitive venue).

What if other reviewers share the same opinion as you? There seems to be a strategic decision-making element involved, which I believe contradicts the fundamental role of a referee. The primary responsibility of a referee is to provide a recommendation, not to influence the editorial decision—that is the editor's responsibility. Therefore, it’s best to focus solely on your own assessment and provide your feedback independently. In many journals, referees are not required to give a recommendation or rating.

I am reserved about the overall idea and have identified some shortcomings that could be improved. The revision provided by the authors addresses some of the shortcomings, but not all. I do not feel that the remaining issues are huge problems, but neither are they negligible. Quality-wise, I believe the paper could be accepted, but I would prefer a rejection.

Why recommend rejection if you believe the paper meets the quality standards for acceptance? If you think the paper makes a significant contribution to the field, it’s better to request a second revision to address the remaining issues.

Answer 5

From what you wrote, it looks like the paper has its value and needs to be published. The only question is whether it is good enough to be published at this venue, or, in other words (and the following is an obvious simplification), if this paper is rejected, will the paper that would be published instead of it be better? You cannot know that. Ideally, the editors can have more information on that, so maybe you should not feel you need to determine the final decision. It looks like the authors' revision improved the paper, but you are still not enthusiastic about the paper. Maybe you should just increase your rating somewhat, as the revision is better than the initial submission, and let the editors do the rest of the dirty work:-).