How many papers should I be reviewing as a referee?

Question

Like many academics with an active research program, I am asked to review many papers. Borrowing an idea from peer-to-peer file sharing companies, I try to keep a rough sense of my recent "ratio" of reviews I have done on other's papers over reviews my papers have received. I try to make sure I have a ratio that is consistently above 1:1. Since many of my papers (and presumably the papers I review) are co-authored, this seems very generous to me.

How many reviews should I be doing? How many is normal? Is a ratio an effective way to keep track of one's reviews? Is a minimum 1:1 ratio appropriate and defensible?

Answer 1

I have wondered about the same question. One thing that I've noticed is that there is an incredible amount of variation in the amount of reviewing done. I have two good friends who are strong, active mathematicians. The last time I spoke to them they were each at least six years past their PhD and had never reviewed a single paper. This seems bad, but it's not their fault: they are not being asked.

I suspect that the lack of coordination among journals in the same field causes the same small number of people to be asked to referee papers over and over again. I am not a journal editor, but I have begun to wonder how editors choose referees: I am someone who is getting (what I think is!) a large number of referee requests. I submit maybe 2-4 papers per year. Not so long ago I used to get that number of referee requests per year, and they were all from people who knew me directly and pertained to topics related to my thesis work or to the areas in which I have written multiple papers. Now I get at least a dozen referee requests a year, and in addition to the above I get some very strange-looking ones. Not so long ago I got asked to referee a paper in (what I think is!) control theory. This was so baffling that I assumed it was some kind of email crossed wires situation and asked the editor about it. It turned out that the paper on real analytic solutions to Bezout equations had been sent to me because of a question on math.SE that I answered, in which I explained why the ring of analytic functions on the complex plane is a Bezout domain. This is distressing: I can see why google would make a connection like this, but in the realm of academic experts, is it not obvious that we are on different sides of mathematics? Experiences like this make me wonder who is refereeing papers.

I'm not sure that counting papers is the right way to gauge whether you are pulling your weight referee-wise. As I mentioned in a comment to another answer, many -- I think most -- papers are refereed by more than one person, either simultaneously for the same journal or sequentially, as many (most?) papers are submitted to more than one journal before they are accepted. Here are some things that I have thought about:

1) It cannot be appropriate for everyone to referee the same number of papers. More experienced people should referee more papers, but also should referee more important / difficult papers. People who have permanent / senior positions in academia (especially: tenure) should have more obligations than those who are struggling to stay in the profession.

2) Not all papers take the same amount of time to referee: not even close, actually. I try to make a point of quickly processing especially short and simple papers: I probably do 3-5 of those each year, and I often process them in a week or even a day. But it would be silly to equate this kind of referee work with the long, difficult papers that you spend weeks or months working on, perhaps having to do considerable outside reading in order to adequately evaluate. Doing 2-3 substantial papers a year is what really keeps me busy as a referee. I suppose I try to keep my eye on the total amount of time I've spent refereeing rather than on the number of papers, although I admit that I don't do this in any formalized way.

3) Unless a paper is clearly of the "one day" variety as above, I try not to let my queue exceed 2 papers. This is also out of respect to the authors: if I have to read two papers before I get to yours, then you'll be waiting for at least six months, probably more. If at any given time every sufficiently experienced/qualified academic had either 0 or 1 papers to referee and made a point of processing papers faster than they submit them, then things would work relatively smoothly, I suppose.

Answer 2

Speaking as an editor and conference organizer, good and reliable reviewers are hard to find. For somebody to be a good reviewer, in addition to being in the right discipline, they must be:

1. A strong enough scientist that I trust their opinion
2. Capable of expressing their analysis of a paper clearly and constructively
3. Good enough at time-management to return the review rapidly
4. Not too overloaded with other responsibilities to accept

Lots of authors fail these tests, especially the first one---anybody who has spent significant time reviewing knows that there are a lot of very bad papers being written. Moreover, papers that are rejected often get revised and sent elsewhere, so they may require multiple sets of reviewers. All of this adds up to the following: if you are a good and trusted reviewer in your community, you will be in high demand, and it's reasonable to have a ratio of greater than 1:1.

For example, in the last three years, excluding grant and tenure case reviews, according to my records I have reviewed 31 papers in 2012, 24 papers in 2013, and 30 papers so far in 2014 (with a few more committed to return before the end of the year). This puts my ratio well above 1:1, with the actual value mainly depending on fluctuations in the denominator.

In sum, I recommend viewing reviewing not in terms of ratios, but in terms of how much time you are comfortable devoting to quality reviews for venues that you wish to support. Don't accept review requests from journals or conferences that you do not respect: there are many low-quality publications that can waste your time. Also, don't accept more reviewing than you can handle, even if that means your ratio is lower: it's better to give one well-thought review than two hasty and shallow reviews that may miss important points and piss off the authors. Do respond yes or no to every review request promptly: otherwise, you are making a problem for an editor who doesn't know whether they need to recruit more reviewers or not.

Answer 3

You could count each n-author paper you are on as 1/n of a paper. Thus each time you submit for publication a 2-author paper, a 3-author paper and a 6-author paper, it's time to referee another paper.

If the journal uses more than one referee per paper, adjust up accordingly.

In some sense this must be what most people do, on average.

Edit: One could ask the editors how many referees they use per paper. Better yet, the editors should volunteer that information.

Answer 4

Four times as many as you submit as corresponding author.

My reasoning is:

• Most journals solicit between two and four referee reports for each paper that is sent out for review. Add to this the time of the volunteer editorial board member who is probably handling the manuscript.
• Some academics do not do their duty to the community and decline all review requests, so the rest of us have to make up for them.
• By my reckoning, this works out to about 4 x the number of papers you submit being a fair load.

Answer 5

First of all, I totally agree with the answer of Pete L. Clark.

However, I do not agree with the mentality that thinks: "Reviewing a paper is JUST doing a favour to the community". We all read papers to stay on top of our fields, get a better understanding of the world around us and to get new insights. I personally read several papers every month. Nonetheless, every paper that I read, I must maintain my critical view. So, there is no harm in reviewing many papers.

When you receive a paper that seems interesting to you, you can decide to review it (of course there is a burden of writing down your thoughts, which is negligible). If the paper is not interesting, you can just decline the review request and move on.