28

Like many academics with an active research program, I am asked to review many papers. Borrowing an idea from peer-to-peer file sharing companies, I try to keep a rough sense of my recent "ratio" of reviews I have done on other's papers over reviews my papers have received. I try to make sure I have a ratio that is consistently above 1:1. Since many of my papers (and presumably the papers I review) are co-authored, this seems very generous to me.

How many reviews should I be doing? How many is normal? Is a ratio an effective way to keep track of one's reviews? Is a minimum 1:1 ratio appropriate and defensible?

21

I have wondered about the same question. One thing that I've noticed is that there is an incredible amount of variation in the amount of reviewing done. I have two good friends who are strong, active mathematicians. The last time I spoke to them they were each at least six years past their PhD and had never reviewed a single paper. This seems bad, but it's not their fault: they are not being asked.

I suspect that the lack of coordination among journals in the same field causes the same small number of people to be asked to referee papers over and over again. I am not a journal editor, but I have begun to wonder how editors choose referees: I am someone who is getting (what I think is!) a large number of referee requests. I submit maybe 2-4 papers per year. Not so long ago I used to get that number of referee requests per year, and they were all from people who knew me directly and pertained to topics related to my thesis work or to the areas in which I have written multiple papers. Now I get at least a dozen referee requests a year, and in addition to the above I get some very strange-looking ones. Not so long ago I got asked to referee a paper in (what I think is!) control theory. This was so baffling that I assumed it was some kind of email crossed wires situation and asked the editor about it. It turned out that the paper on real analytic solutions to Bezout equations had been sent to me because of a question on math.SE that I answered, in which I explained why the ring of analytic functions on the complex plane is a Bezout domain. This is distressing: I can see why google would make a connection like this, but in the realm of academic experts, is it not obvious that we are on different sides of mathematics? Experiences like this make me wonder who is refereeing papers.

I'm not sure that counting papers is the right way to gauge whether you are pulling your weight referee-wise. As I mentioned in a comment to another answer, many -- I think most -- papers are refereed by more than one person, either simultaneously for the same journal or sequentially, as many (most?) papers are submitted to more than one journal before they are accepted. Here are some things that I have thought about:

1) It cannot be appropriate for everyone to referee the same number of papers. More experienced people should referee more papers, but also should referee more important / difficult papers. People who have permanent / senior positions in academia (especially: tenure) should have more obligations than those who are struggling to stay in the profession.

2) Not all papers take the same amount of time to referee: not even close, actually. I try to make a point of quickly processing especially short and simple papers: I probably do 3-5 of those each year, and I often process them in a week or even a day. But it would be silly to equate this kind of referee work with the long, difficult papers that you spend weeks or months working on, perhaps having to do considerable outside reading in order to adequately evaluate. Doing 2-3 substantial papers a year is what really keeps me busy as a referee. I suppose I try to keep my eye on the total amount of time I've spent refereeing rather than on the number of papers, although I admit that I don't do this in any formalized way.

3) Unless a paper is clearly of the "one day" variety as above, I try not to let my queue exceed 2 papers. This is also out of respect to the authors: if I have to read two papers before I get to yours, then you'll be waiting for at least six months, probably more. If at any given time every sufficiently experienced/qualified academic had either 0 or 1 papers to referee and made a point of processing papers faster than they submit them, then things would work relatively smoothly, I suppose.

  • If you are submitting 4 papers per year and getting at least 2 reviews for each paper, then 12 papers is about the amount that you need to be reviewing to pay back the reviewing and editing time that is being put into your submissions. See My answer below for more on why. As to how editors select reviewers, we look for reviewers who are already in our database and have keyword matches with the submitted paper, or else we look for people who have published on similar topics in recent years. If your reviews are good and timely, you will be asked more often. – Significance Feb 26 '16 at 8:20
  • 1
    I think that point 2 is an excellent one: simply counting the number of papers reviewed is pointless. Measuring the time spent on the reviews is much more meaningful. – DCTLib Feb 26 '16 at 9:29
12

Speaking as an editor and conference organizer, good and reliable reviewers are hard to find. For somebody to be a good reviewer, in addition to being in the right discipline, they must be:

  1. A strong enough scientist that I trust their opinion
  2. Capable of expressing their analysis of a paper clearly and constructively
  3. Good enough at time-management to return the review rapidly
  4. Not too overloaded with other responsibilities to accept

Lots of authors fail these tests, especially the first one---anybody who has spent significant time reviewing knows that there are a lot of very bad papers being written. Moreover, papers that are rejected often get revised and sent elsewhere, so they may require multiple sets of reviewers. All of this adds up to the following: if you are a good and trusted reviewer in your community, you will be in high demand, and it's reasonable to have a ratio of greater than 1:1.

For example, in the last three years, excluding grant and tenure case reviews, according to my records I have reviewed 31 papers in 2012, 24 papers in 2013, and 30 papers so far in 2014 (with a few more committed to return before the end of the year). This puts my ratio well above 1:1, with the actual value mainly depending on fluctuations in the denominator.

In sum, I recommend viewing reviewing not in terms of ratios, but in terms of how much time you are comfortable devoting to quality reviews for venues that you wish to support. Don't accept review requests from journals or conferences that you do not respect: there are many low-quality publications that can waste your time. Also, don't accept more reviewing than you can handle, even if that means your ratio is lower: it's better to give one well-thought review than two hasty and shallow reviews that may miss important points and piss off the authors. Do respond yes or no to every review request promptly: otherwise, you are making a problem for an editor who doesn't know whether they need to recruit more reviewers or not.

11

You could count each n-author paper you are on as 1/n of a paper. Thus each time you submit for publication a 2-author paper, a 3-author paper and a 6-author paper, it's time to referee another paper.

If the journal uses more than one referee per paper, adjust up accordingly.

In some sense this must be what most people do, on average.

Edit: One could ask the editors how many referees they use per paper. Better yet, the editors should volunteer that information.

  • 2
    I don't know of any evidence that suggests that this is what people do on average. My sense is that some people review tons of papers and many people (e.g., graduate students) submit papers but are rarely asked to review. – Benjamin Mako Hill Sep 27 '14 at 22:07
  • 9
    @BenjaminMakoHill: On average, people do exactly as many reviews as they receive (scaled by the number of coauthors). Every review that someone receives comes from someone. – Jukka Suomela Sep 27 '14 at 22:36
  • 2
    The averages must be exactly the same if we average over all people who have ever submitted or reviewed a paper, but not necessarily over people currently active in the field. (For example, it's possible that people who leave the field have on average submitted more papers than they have reviewed.) – Anonymous Mathematician Sep 27 '14 at 22:57
  • 5
    (Just to clarify. When you review a paper, the math goes like this: "this counts as 1 review written by me". And when you submit a paper with n coauthors and receive the reports and see that there are k reviews, the math goes like this: "this counts as k/n reviews received by me". You do the same thing regardless of whether your paper was rejected or accepted. There is no need to get any additional information from the editors.) – Jukka Suomela Sep 28 '14 at 7:14
  • 5
    But I agree with the gist of what you're saying: trying to compute the average is needlessly complicated. You can probably get a pretty good idea of how many referees you "use" per year and make sure that you stay ahead of that. How far ahead you want to be is still a judgment call, as I talk about in my answer. – Pete L. Clark Sep 28 '14 at 8:15
0

Four times as many as you submit as corresponding author.

My reasoning is:

  • Most journals solicit between two and four referee reports for each paper that is sent out for review. Add to this the time of the volunteer editorial board member who is probably handling the manuscript.
  • Some academics do not do their duty to the community and decline all review requests, so the rest of us have to make up for them.
  • By my reckoning, this works out to about 4 x the number of papers you submit being a fair load.
  • 1
    "Most journals solicit between two and four referee reports for each paper that is sent out for review." This is highly subject-dependent. In my field -- mathematics -- the most common number of referees is one. I have submitted over 30 papers for publication and never gotten more than two. – Pete L. Clark Feb 26 '16 at 9:43
  • 3
    By the way, if you want to be quantitative about it (which, as I say in my answer, I don't necessarily think is the best way to go), then how is the number of coauthors taken into account in your reckoning? – Pete L. Clark Feb 26 '16 at 10:02
  • The number of co-authors is irrelevant if you count only the number of papers that you submit. You, personally, as corresponding author. I'm really surprised to hear that one review is sufficient in some fields. That must speed up the review process a lot, and make responding to review comments much easier (no navigating conflicting reviewer opinions)! The journal for which I am an editor prefers 4 reviews and requires at least 3 for each manuscript. – Significance Feb 26 '16 at 22:03
  • "That must speed up the review process a lot, and make responding to review comments much easier (no navigating conflicting reviewer opinions)!" Well, one frequently gets referee reports in less than a year, but not always. I hope, for your sake, that that doesn't sound that fast to you. – Pete L. Clark Feb 26 '16 at 22:21
  • I missed the "corresponding author" part of your answer, in part because it didn't appear in your comment. No one I know would interpret "I submit 2-4 papers per year" as "I submit 2-4 papers per year as corresponding author," since in my field "corresponding author" means precisely "the person who sends the email to the journal / fills out the online form". I know of no reason not to divide corresponding author duties roughly equally among authors, but on the other hand I have no guarantee of it. Some people are undoubtedly corresponding authors more or less often than their coauthors. – Pete L. Clark Feb 26 '16 at 22:25
0

First of all, I totally agree with the answer of Pete L. Clark.

However, I do not agree with the mentality that thinks: "Reviewing a paper is JUST doing a favour to the community". We all read papers to stay on top of our fields, get a better understanding of the world around us and to get new insights. I personally read several papers every month. Nonetheless, every paper that I read, I must maintain my critical view. So, there is no harm in reviewing many papers.

When you receive a paper that seems interesting to you, you can decide to review it (of course there is a burden of writing down your thoughts, which is negligible). If the paper is not interesting, you can just decline the review request and move on.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.