Dealing with rejection anxiety as a reviewer

Question

I am not very experienced as a reviewer, and I often find myself stressing over recommending rejection, even for the articles which I think are clearly low-quality and unsuitable for publication in a strong journal. I imagine how the authors might feel (especially if I suspect they are PhD students/postdocs/early career researchers who have a lot at stake in the publication), and I second-guess my judgement and wonder if it would align with others’ opinions. I also always worry that I might come across as unnecessarily harsh and critical in my reviewer report, especially as English is not my native language, so I have limited ability to express myself clearly and succinctly.

This usually leads to me writing too verbose and too detailed review reports (I think). More critically, it caused me a few times to recommend major revisions in the first review round when I thought that the manuscript should really be rejected but didn’t have the guts to suggest it. So inevitably, there comes the second round, the manuscript is not much improved, and I am left stressing again over recommending rejection. I think in such instances it would have been better to just suggest rejection initially, and more importantly it would probably have been fairer to the authors and would not waste their, my or the editor’s time.

Basically, I am asking how to overcome this anxiety about suggesting rejection and the fear of coming across as the “bad reviewer.” Are my concerns valid?

Answer 1

The role of the reviewer is not so much gate-keeper, but rather your role is to help the authors get the paper to a position where it is publishable. So I very rarely recommend outright rejection at the first round.

Once you have written things down what it would take for the paper to be publishable, it maybe the case that this is clearly unachievable and that this author will never get this paper to that level – but that is for the editor and authors to decide.

Reviewers are normally asked to judge two things:

1. Is the paper sufficiently novel and exciting to be interesting to a large enough fraction of the readership of the journal.

2. Is the paper sufficiently rigorous to demonstrate the claims it makes.

If the paper is not sufficiently novel or exciting I say:

To make this paper publishable, the authors must explain the additional advances made compared to Smith et al. and Jones et al. and why their work would be of interest to a reader of the Journal of Things, beyond those connected with the sub field of very small things.

Methodological or interpretive flaws I will simply point out and say they must be fixed.

So for example, I recently had to review a paper for a software tool. It unlikely to get published, but instead of saying “reject” I said:

1. The tool does not appear to implement any new functionality compared to Jones et al and indeed is missing several of the functions in that tool. The authors should explain why their tool is an advance in comparison, and provide benchmarking data comparing the performance.

2. The tool was not installable on my system, the authors should ensure that there is a robust installation routine and that it functions on representative range of systems. {goes on to list the problems with the installation}

3. The tool has several bugs and conceptual errors in it that must be fixed before it was published. {goes on to list them}

In conclusion, the tool has several usability and functional shortcomings that must be fixed. Even if these were fixed, the authors must explain why their tool is superior, or even equal in performance or features to Jones et al.

Now I was 80% sure that this wasn’t going to happen – the amount of work required to bring the paper up to a publishable standard was probably more than had been put into the project up until that point – but if they wanted to do it, I’d be happy to recommend publication. I never saw another revision of the paper – I don’t know if the editor decided to reject it, or the authors decided it wasn’t worth it –, but I fulfilled my role: to say what it was that was necessary to bring it up to standard.

If the authors do decide to try for a resubmission, without fixing the problems I identified, I will just say that:

The authors have not addressed points A, B and C, and I cannot recommend publication until they have done so.

---

There are several "Peer Reviewer's Oaths" out there. The one I like is from here:

http://www.opiniomics.org/the-reviewers-oath/

I, the reviewer, promise:

1. to not hide behind a screen of anonymity
2. to be open and honest with you (the authors) at all times
3. to be constructive in my criticism
4. within the rules given to me by the journal, to assist you in every way I ethically can to get your manuscript published, by providing criticism and praise that is valid and relevant

Answer 2

Basically, I am asking how to overcome this anxiety about suggesting rejection and the fear of coming across as the “bad reviewer.”

For me, one part of it is fairness: I assume that most reviewers act according to their best judgement, and that includes recommending rejection for papers that clearly need to be rejected. If I allowed my judgement to be affected by what is essentially an ego issue (not wanting to be "the bad guy"), I would introduce unfairness against other authors who get more objective reviewers.

This usually leads to me writing too verbose and too detailed review reports (I think).

This can be fixed by organizing your review in a readable way. Start with an itemized list of the main arguments for and against acceptance (only bullet points or a brief sentence per item). Then explain the points from this itemized list in more detail. Specifically, explain what needs to be fixed for the paper to become acceptable. Only after this "high level" discussion, give a full discussion of all the details (most of which will probably not affect the acceptance decision). For typos and grammar issues, have a separate list at the end.

More critically, it caused me a few times to recommend major revisions in the first review round when I thought that the manuscript should really be rejected but didn’t have the guts to suggest it. So inevitably, come the second round, the manuscript is not much improved and I am left stressing again over recommending rejection.

This would be fixed as well if your first review clearly specified the conditions which need to be fulfilled to make the paper acceptance-worthy. If the authors submit a version that doesn't fulfill the conditions, then it's the authors' fault, and you don't need to stress out.

Answer 3

You are not rejecting, you are recommending. The editor does the descision and there should be a second referee. So don't sweat it - be honest.

Answer 4

It is nice of you to think about the author of a paper you feel it should be rejected. Please also think about:

• All the readers. They waste their time reading a bad paper. They were excited when they found the paper but disappointed after reading it.
• The authors of other papers, that might be rejected (due to limited number of presentation slots or restricted number of article per issue).
• And the reputation of the conference / journal that might be compromised by bad papers. This is also steered by the editor, but he has to rely on hisreferees. (thanks to Gnudiff for this aspect)

Answer 5

There are quite a few things to unpack here, so let’s go over them one by one:

Career Damage

You are responsible for delivering an accurate report; you are not responsible for indirect consequences of an accurate report, such as career impact. If somebody does not get a position because you gave a correct negative assessment, this is upon them or their co-authors. Also remember that somebody else will get the respective position instead and – on average – they are more qualified: There is a lot of noise in career decisions and by providing a fair review, you reduce this. Do not only think of the authors, but also think of the qualified researcher who does not get a position because their opponent managed to get some piece of junk published thanks to overly lenient reviews.

If you want to be nice to upcoming researchers without compromising the quality of research, do your peer reviews quickly (i.e., prioritise them with respect to your other duties, not rush the actual reviewing). This way, they have more time to fix the paper’s problem and submit it elsewhere before whatever application deadline they are facing and they hopefully learn something in the process:

Coping with and learning from rejections and criticism in general is an important skill for researchers. If a student’s supervisor did not properly guide them through their research or writing process, the student has to learn this another way. You are not responsible for taking that part of the training, but a detailed review can be that way.

Finally, it may also help you to assume that all the authors are professors close to retirement with no career stakes whatsoever. The ideal of peer review is that it should ignore the authors’ identity (which is why double-blind peer review is a thing).

Assessing Unsalvageability

From the reviewer’s and editor’s point of inevitably imperfect knowledge, every paper has the potential to be salvaged: The authors may fix their research or explanations or present better arguments for the relevance of their work. The probability that a paper can be made suitable for a journal is never zero. Going by this, there never should be a decision for final rejection. But this is not what a rejection practically means.

Instead, if this probability (of the paper eventually being made suitable for the journal) turns out to be below some implicit threshold, it is more feasible to reject the paper and not invite further submissions, which would waste further journal resources. All you do is to help estimate this probability and whether it is below the threshold. If your estimate is a little bit off, that’s not dramatic, as there are several mechanism to compensate for this, namely other reviewers, the editors, and the option for rebuttal, if the authors think that they solved the issues with the manuscript against all odds.

What’s more important is that you elaborate what problems you see with the paper and what needs to be done to solve this (Ian Sudbery’s answer gives a good example for this). Even if your criticism should be based on some misunderstanding, this allows the authors to clarify. Nothing is more frustrating if the main argument for rejecting your paper is something like:

I don’t understand Section 2. [No further elaboration]

as you have to resort to guessing on how to solve this. Which brings us to your next concern:

Too Detailed Reviews

I do not think details are harmful in a review per se. The main reason to stop delving into details is to avoid wasting your time. Of course, you should make clear what the main issues are that lead to the rejection. A detailed review can be very helpful to the authors to solve these main issues, be it for a resubmission or submission to another journal. Moreover it provides evidence to the editor that you seriously engaged with the manuscript thus giving your review deservedly more weight. Keep in mind that almost every manuscript gets published eventually, so your effort is rarely wasted (unless you do such things as noting typos in paragraphs that need to be completely rewritten anyway).

Rejection Rate

You may note that you recommend to reject papers more often than your own papers are rejected or you even hear of papers being rejected. There is a good chance that this says more about the quality of your and your acquaintances’ submissions than about your recommendations as a peer reviewer. Remember Sturgeon’s law: “90% of everything is crap.”

It may be worth checking out the actual rejection rate of the journal if you can find it. But even if your rejection rate is higher than the general rejection rates of the journals you are reviewing for, this is not necessarily a bad thing. First of all, there is the matter of chance: You have to review quite some papers to make any substantiated statements about your reviews deviating from the mean. Second, the more thorough a review is, the more likely it is to find a flaw leading to a recommendation of rejection. Here, the problem is not necessarily you but other reviewers not being thorough enough to spot problems.

Wording

All reviews I have received, written, and co-written were almost exclusively in a very neutral tone, i.e., neither overly euphemistic nor aggressive. There are justified exceptions, e.g., I once wrote in a report that I was disappointed by a revision on account of not implementing straightforward suggestions with which the authors appeared to agree. If you stick to a neutral description of issues and suggestions, I see little that could go wrong.

The one exception where wording matters is the actual recommendation – if this happens in writing and not via a score system. Here, you do not want to be misunderstood by the editor. The aforementioned list of things that need to be happen before acceptance is a good way to avoid any miscommunication here: For example, if the editor can easily see that this is probably an insurmountable challenge, they do not need your direct assessment. It may also help you to write a short note exclusively for the editor explaining how likely you think that the authors manage to fix their manuscript.

Answer 6

I think the roles of "major revision" and "reject" are quite different, and it is not helpful to the author to use one when you really mean the other.

Reviewers should recommend rejection if the paper, while it may be publishable, is not sufficiently interesting for the journal. If this is the case, it is unlikely (at least in my field, which is mathematics) that a revised version would ever be published in that journal. However, the paper may well be suitable for a slightly less selective journal, and you do authors no favours if you ask for a major revision first, rather than giving them an early opportunity to resubmit to such a journal. Ideally your review would suggest alternative venues (as well as identifying any significant issues they may wish to deal with before submitting).

Major revisions, on the other hand, are appropriate for papers which have issues which need to be resolved before publication (and it may not be clear that the authors will be able to resolve them), but with an interest level suitable for the journal.

Answer 7

1. "The best lack all conviction". Your doubts sound more symptomatic of you doing a good job refereeing than of you being unnecessarily critical. I do think it's a good idea to give people a chance to improve their paper with a "major revision" recommendation. Keep in mind that a recommendation is not a verdict (the editors base their decision on your report, but this doesn't mean that they just adopt your suggestion without change; in particular, a "major revision" can become a "rejected" if the other referee is similarly unenthusiastic).

2. Here is how I would sanity-check my "major revision" suggestions: Assuming that the author followed all my recommendations, would I then suggest accepting the paper, or would I be annoyed at having to find better reasons to reject it? In the former case, "major revision" is the right recommendation to make. In the latter, it is "rejection". (And then there is "revise and resubmit", which I tend to choose if I don't understand or don't believe significant parts of the paper and but wouldn't be surprised if the author salvages it. Here I am deferring judgment until the problems are resolved. This option is good to keep in mind when you don't quite see what the paper will become after revision.)

3. "Too detailed" is rarely an issue in reviews. Even if the paper cannot be salvaged, any future papers by the same author will probably gain from you pointing out grammatical errors, confusing terminology, misconceptions, etc. (which otherwise would likely creep over into these future papers if left unchallenged). Peer review is one of the few ways scientists learn after their PhD!

Answer 8

Speaking as an editor, let me reiterate something you know: the decision to reject or not is the editor's, not yours. You are asked for a recommendation, but the editor (who typically has much more experience) makes the decision. The contents of your report matter more than the recommendation. As an editor, a big part of my job is recognizing the "major revision" recommendations that should be interpreted as "reject" (and less often, vice-versa).

Some great detailed advice for refereeing can be found here.

Answer 9

In the long term, you need to get more experience to feel more confident about your decisions. So what you can do now, is that you can try to make this process faster by 1. Getting familiar with the state-of-the-art on the topics you are invited to review 2. Volunteer for more reviews 3. Study other people's reviews (as a reviewer, at the end of the process you can see other reviewers' comments and also, some venues have openreviews (look it up), this will tell you how you are doing in comparison to others e.g. are you too optimistic? 4. Write more papers, which helps you get the hang of it

Mindset wise:

1. Reviews should be written in a way to suggest how the paper could be improved, and why, and point out major conflicts/false statements etc.. So, no, you don't need to bash a paper. There are far too many reviewers that think this is the point of reviews, which is shameful really. You should keep as objective as possible and not let your "subjective" judgement get in the way. Constructive criticism is what you should aim for.

2. If a paper is indeed too bad, by giving them more chances you are actually skewing the reviewing process because not all are lucky to have reviewers who will just give it a go.

3. If you are in doubt though, give the benefit of doubt to the authors. I have received far too many false claims that I couldn't even answer to in rebuttal, or did but the reviewers mind was set from the beginning. Just because I didn't include a detail, assuming that I didn't do it that way is still just subjective judgement (in fact, I did this but to me it was trivial and standard for any research paper so I didn't flat out write the exact details for example).

All that being said, I am also on the more optimistic side and give out more positive reviews with a similar mindset, but I also realize that when I am assigned perfect matches, I can be crystal clear in my comments and judgement, so you just need to be more confident in your expertise.

Answer 10

I think it's important to remember that your role as a reviewer isn't to make value judgements about the authors' work. By recommending rejection, you're acting more like a moderator. You are interpreting the rules — the written rules of the journal or publication, and the unwritten rules of peer review — and applying them to a given paper. You are answering the question “Does this paper adhere to the rules and standards (or can it with modifications)?”, and not the question “Is this a good paper?”.

Recommending rejection is then just answering the question in the negative — you do not believe this paper adheres to those rules, and isn't likely to after modification. It's not a “bad paper” (well, it may be, but you're not saying so), and the authors are welcome to find a different venue that better aligns with what they've written.

Finally, remember that you are not the only reviewer. If you recommend rejection and the paper is rejected, then at least one other reviewer or the editor agreed with you. And if you recommend rejection but the paper is accepted, no one will give another thought that you recommended differently.