32

I made a little statistics about my review decisions. I found out that I reject way more papers than I accept. I honestly try to accept the reviewed papers, I read them carefully and my review are always long. But most papers I review are low quality (in my opinion). Some colleagues told me that I should be more gentle in my review, as this will affect my profile somehow (negatively of course). Is that true ?

Also: should I adapt the toughness of my review to the quality of the journal/conference I am reviewing for?

  • 1
    Do you happen to also have a statistics how many of the papers you had rejected have been published eventually? – Wrzlprmft Apr 11 '14 at 23:02
  • @Wrzlprmft .. I didnt keep track. I know a very poor one got accepted though. – AJed Apr 12 '14 at 17:10
23

It is of course impossible to say if you are overly critical. But, some statistics might help. With prestigious journals the rejection rate might be 90% (very high in any case). I am Editor in Chief of a smaller journal and we have a steady 50% rejection rate. The extremely high rejection rates in some journals come from limited space, in my case we simply impose strict guidelines and the rejections come from reviewers assigned by associate editors. So the rate in our case is fairly certainly upheld by what the community sees as acceptable or not.

So if you end up finding you reject large numbers (at least around 50%) I would not say that it is out of the ordinary. I do not of course know what typical rejection rates are in the journals for which you do reviews, so the number may of course vary.

So getting to the question of how this affects you. I very much doubt that you get a negative effect by being a fair reviewer. If you do many reviewes over time and you are not considered fair, I am sure you would notice the numbers of appointments dropping. If you get re-invited by the same journals/editors, I think they appreciate your work and I would take that as a sign that you do fine.

Should you try to adapt to the journal? In general, no. You should apply your knowledge and understanding to provide a critical review of the submitted work. An editor should take your comments along with comments from one or more additional reviewers and provide a recommendation to the authors. You are not solely responsible in this process! If the journal is such that it publishes material with which you are familiar but where its audience is in a different field, you may consider handling the paper differently, but obviously not ignoring errors and misconceptions.

16

An important thing to be aware of, is that (as far as I can judge) editors tend not to send papers on which they have a bad first impression to experienced people. It follows that junior researchers tend to receive a relatively large proportion of demands to review papers that are below average. That might explain your feeling.

  • 3
    Thanks, this explains a lot in fact (I cant consider myself senior). I wonder though if professors still review papers. I feel most of them send it to their students. – AJed May 20 '13 at 23:54
13

Many more papers are submitted than accepted at top venues/journals, so statistically speaking, you should reject more than you accept.

Harsh reviews are ultimately good for science, so long as they filter the crap and help improve papers with potential but are not there yet. If your review is able to help the PC/editors decide whether or not to accept the paper and help the authors improve their paper, then you are doing your job correctly. If you are harsh but unhelpful, then you need to change your approach.

I think your reviews can change a bit depending on the venue. If you are reviewing for a workshop, your goal would be to help choose papers that are likely to generate interesting discussion. This is especially the case if there is no associated official publication.

  • 1
    "Many more papers are submitted than accepted at top venues/journals, so statistically speaking, you should reject more than you accept." Yes, but at many journals editors have the discretion to reject papers without refereeing them. So your statistical conclusion does not necessarily follow. – Pete L. Clark Apr 12 '14 at 22:31
5

Some journals allow you to see the reviews provided by the other reviewers of the paper, which is often a good way of monitoring your own performance as a reviewer. If you are harsh to an unreasonable degree, I suspect that editors will stop sending you papers to review, so if they are still sending them, that is an indication of sorts that there is not a fundamental problem.

As long as you are happy that you are applying a standard that you feel is reasonable, I'd stick with it. You probably review too few papers to know whether you are being unduly harsh or that you have just been sent a series of bad papers by random chance.

When I get a rejection, I am generally fine provided that the reviewer does a good job of explaining what they (incorrectly ;o) feel the problem is and what needs to be done to the paper for them to be happy with it, in sufficient detail for me to be able to do so if it is actually possible. The problem I have with negative reviews is when they make vague criticisms that can't be addressed as there is not enough information to actually act on, or where reviewers raise additional issues on the second review to reject your paper that could have been raised on the first review. So if your reviews are detailed and constructive, they are probably good reviews.

At the end of the day, it is ultimately in the author's best interests not to publish poor work, and if it can be usefully improved, then the reviewer is doing them a favour in not accepting it.

4

In addition to points made in other answers, and strengthening Dave Clarke's first sentence: not only the best, but even "medium-good" journals receive far more good submissions than they could possibly publish ... at least within the traditional "page limit" constraints, and possibly within some tacit constraint about appearing sufficiently selective. (That is, even if all the papers were quite good, a journal that publishes 10,000 pages a year will be suspect...)

Thus, the job of "editor" tends to degenerate into trying to find reasons to reject.

There simply are not enough "prestigious" journals to publish all the (good!) work done by all the good young people... who now can typeset things much faster and get them submitted. (In contrast, pre-TeX, pre-internet, getting something sufficiently presentable to send to a journal was much more of a hassle... Also, the number of researchers has grown much faster than the number of good journals, at least in my field, mathematics.)

Returning to the literal question: as others have said, indeed, try to be constructive and helpful, rather than merely a gatekeeper. At the same time, you cannot count on corrective feedback from editors if you are "too harsh", because they may not realize this, and in fact have incentive not to worry about that exigency, because you are making their job easier.

Also, one should note that in recent times many respectable journals explicitly declare to referees that they, the referees, are not responsible for correctness of an article. It is supposedly the author's responsibility. This is obviously strange, but can be understood when we realize that the gatekeeper function of journals, editors, and referees is at least as important as the scientific function... as disillusioning as that might be. But it does explain situations that are otherwise baffling and counter-intuitive.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.