36

I never tried to avoid becoming a sub-reviewer, and still, this question made me wonder. How much time should I, as a PhD student, spend on a review of a paper that my advisor gave me to do? Does the answer depend on whether I'm getting credit for the review or not, that is whether I'm a sub-reviewer or not?

Improve this question

5 Answers 5

Reset to default
40

You should do slightly more reviewing work for the community than the community does for you. So, as a rough estimate, assuming every paper you write and submit needs three reviews and your average paper has three authors, you should review slightly more papers than you submit.

There is no way to predict in the abstract how long a paper takes to review; that varies from field to field, subfield to subfield, and even paper to paper. Answering a similar question in Theoretical Computer Science Stack Exchange, I wrote:

Expect to spend about an hour per page, mostly on internalizing the paper's results and techniques. Be pleasantly surprised when it doesn't actually take that long. (If it takes significantly less time than that, either the paper is either exceedingly elegant and well-written, you know the area extremely well, or the paper is technically shallow. Don't confuse these three possibilities.)

No, the answer does not depend on whether you get credit for the review. If you're not going to write a thorough, professional review, just say no.

Improve this answer
6
  • 9
    +1 for not accepting the review if you do not intend to invest the time needed.
    – Paul Hiemstra
    Commented Dec 16, 2012 at 20:58
  • 11
    Note that you must calculate based on papers submitted and reviewed not papers published.
    – Rex Kerr
    Commented Dec 17, 2012 at 16:28
  • I fully agree with this answer. @JeffE
    – Coder
    Commented May 9, 2016 at 14:27
  • 1
    A very nice answer, except that if your advisor gives you a subreview, most likely you can't "just say no". Your advisor expects a review.
    – Caleb Stanford
    Commented May 29, 2018 at 23:44
  • 1
    @StrongBad Closer to the former. A quick glance at my own papers suggests that preprints, including conference and journal submissions, typically have about 600-750 words per page; this is probably the most common format that people actually read. Single-column mathy-journal papers have about 500 words per page; two-column proceedings papers can have over 1000 words per page. Nobody double-spaces anything.
    – JeffE
    Commented Dec 21, 2018 at 15:37
12

Read the entire paper in one go. For a 12 page* paper, that should take you an hour at most. This is the first pass. From this you can now decide how to proceed, now that you have (quickly) covered all the sections. (edit: The goal on this first pass is not to understand all the details, but rather get an overall feeling of the paper, quality and organization of ideas)

If a paper is terribly written and difficult to understand you should reject it outright for being so. It should not take a reader 12 hours to read a 12 page paper - and indeed, many papers are badly written and do take this much time - and you are still left wondering because it was not clear.

If the basic claims seem sound and sufficiently interesting/important and the idea is clear enough to understand, then you need to dig in. Depending on the type of paper this means looking at definitions, proofs, or experimental design, hypothesis testing and the analysis of the results. Likewise for any related work you know of that should be cited (and not just your own - that is too cliche! - if you do recommend your work as a citation at least include some others!).

All along, help people out with typos and readability. As a reviewer, you are the last chance to make this a good read before it is published.

And even if you reject it, don't just shoot people down. Give them ideas about how to make the paper stronger, suggestions for different directions, etc. Remember that it is most likely some other student who will receive your review back - so try and help them make the next paper better.

(* 12 single column pages. A 6 page paper with two columns can often contain the same amount of text as a 12 page single column paper)

Improve this answer
5
  • 11
    Your answer is somewhat field-oriented. Very few 12-pages papers in my field (fundamental mathematics) can be read in one hour (except if you mostly read the introduction), at least not by me.
    – Benoît Kloeckner
    Commented Dec 17, 2012 at 12:24
  • 2
    I am also a mathematician, and I agree with Benoit. One can easily (and profitably) spend more than one hour per page on a math paper. On the other hand, there are other 12 page papers that I have spent only a few hours reviewing in total...and have recommended for acceptance. This makes me think that it is not helpful to suggest figures on hours per page.
    – Pete L. Clark
    Commented Dec 17, 2012 at 16:52
  • 2
    In my field, computer science, and especially papers with theoretical proofs and algorithms, it is the same. I still think that a full first pass of the paper in an hour is a good approach for the following reason: if the paper cannot compel you to dig deeper with a single pass at 1 hour, how do you expect any other reader to actually read this paper. The first pass should entice you into going deeper - if it does not because it is badly written or makes a trivial claim, then it not worth going further.
    – Tom Carchrae
    Commented Dec 17, 2012 at 18:47
  • 1
    @Tom: sure, taking about an hour to flip through a 12 page paper sounds quite reasonable. I was objecting to your sentence "It should not take a reader 12 hours to read a 12 page paper". Why not?? Or are you still talking about skimming when you say this?
    – Pete L. Clark
    Commented Dec 20, 2012 at 18:52
  • I agree with your previous comment - there is no magic number on the number of hours you can spend reading a paper, especially one that is several pages of proof. That said, I still feel that it is an author's obligation to make the salient points clear enough for a reader to understand the big idea quickly! As to the upper bound on time taken to digest the details of a paper, that could take a lifetime. In the case of a review, I would only progress to this level(not a lifetime) of detail if the big idea was clear and significant.
    – Tom Carchrae
    Commented Dec 20, 2012 at 21:41
9

There is no clearcut answer to how long a review takes. I would say that the amount of time spent on it is the time you need to understand the paper and provide good advice on wether or not it is a paper that can be accepted, with some changes or as is. In general, this should take you around a full working day. How long this exactly becomes depends on:

  • The level and clarity of the paper (very bad paper takes less time, mediocre paper takes more time, very good paper takes less time).
  • Your familiarity with the subject (less familiar more time).
  • The requirements of the journal/conference you are reviewing for, is it a full review, or just a go/no go.
Improve this answer
5

It depends on how many reviews you've done and what your experience is. If you're just starting out and have never reviewed a paper before, you should expect it to take a minimum of several hours to do a decent to good review.

Your mileage may vary depending on what discipline you're in, but a good review generally will consist of:

  • a summary of what the contribution of the paper was
  • a brief list of 1-3 strengths and 1-3 weaknesses that the paper had
  • several paragraphs explaining those strengths and weaknesses in greater detail along with constructive suggestions of what would be needed to improve the weaknesses
  • a brief listing of any related work that you feel is missing from the paper
  • a recommendation to the reviewing committee of whether or not you believe the paper should be accepted
  • usually there is a part for a numerical evaluation of your expertise and whether or not it should be accepted

The more reviews that you do, the less time that you should need to spend on doing the review. It seems like many professors can knock out a review in an hour or less, but most of the graduate students I've seen will tend to spend at least 1-2 hours per review if they're doing a good job.

Improve this answer
4
  • 5
    Same comment than to Tom Carchrae answer: your time indications must be somehow field oriented, because I cannot see how one would do even a barely decent review in one or two hours in my field (where typical reviews come back six months after submission, due to the time it needs from the referee).
    – Benoît Kloeckner
    Commented Dec 17, 2012 at 12:27
  • 5
    Why would you limit the review to 1-3 weakness? Surely you want to point out all the weaknesses.
    – StrongBad
    Commented Dec 17, 2012 at 13:10
  • Again, your mileage may vary depending on what discipline you're in, but my understanding has always been that you want to try to group weaknesses into a few main points in a list, then to expand on all of the things that are wrong in the paragraphs under the headings of those topics so that it's easier for the authors to understand your criticisms and to address them.
    – Zai
    Commented Dec 17, 2012 at 13:53
  • 7
    I think it is on the border of irresponsible to suggest that a review of a paper can or should be done in an hour or less.
    – Pete L. Clark
    Commented Dec 17, 2012 at 16:56
3

Take enough time to find most (~80%) of the important strengths and weaknesses of the paper and to describe and explain the weaknesses thoroughly and clearly but if you encounter one particular non-core aspect or section of the paper that would take excessively long to do this, just skip it and be frank: "I have not reviewed ((aspect X)) thoroughly, because my expertise in this area is too shallow."

If the editor is worth her salt, this will actually increase her confidence in the rest of your review.

All of your reviews should be careful but none of them need to be perfect.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .