How long typically are paper reviews? Is there such a thing as “too long”?

Question

When you review papers submitted for publication, is there an “optimal” length for reviews? In my experience as an author and referee, I have seen a large range of review lengths (for reference, a paper in my field is typically between 3 and 8 printed pages):

• Zero length: for some of the papers I have authored, the reviewer just clicked the “accept as is” or “reject” checkmark on the review sheet, without adding any comment (at least, not any comment visible to me). It's not so common, but it has happened.
• Short length: a lot of time, I received reviews who consisted of a single paragraph.
• Medium: one full page, maybe two pages.

Although I have never received any such long reviews, I have myself written on a few occasions reviews that exceed two pages, including once or twice a four or five-page review. These were cases where the paper was good, but could be (in my view) much improved and some of the aspects/consequences had escaped the authors' consideration.

I am wondering how useful reviews of various length are to editors. I often consider that “unmotivated” reviews are useless, as they do not give any real insight about the paper to the editor. For example, if the editor gets two conflicting non motivated reviews, how is she to decide?

On the other hand, although I write some from time to time, I have never received long reviews, so… is this something frowned upon?

Answer 1

Reviews can be of quite varying length but obviously the extremes indicate some problems.

A review consisting of "Accept as is" would be highly suspicious in my mind (as an editor). It usually means the reviewer has not done any work, essentially no manuscript is that close to perfect (although it may of course happen). A review of "Reject" without additional comment is equally pointless (I am then assuming the journal has some form of quality check before accepting for review). An absence of comments is just a big warning sign since there is no perspective on why the MS is either perfect or perfectly worthless.

Considering the length of a review, it is governed by two factors: the quality of the manuscript and the personality of the reviewer. To some extent longer reviews indicate more questions to be resolved. At the same time some reviewers may be more nit-picking than others so that also influences the length. Based on my experience as an editor, I would say, as a rule of thumb, that at least a page of (single spaced) comments would be a basis for a descent review for a normal manuscript (15-20 pages double spaced excluding references, tables, figures) in the field experiment/observation based science where I work. A review of more than three or four pages of (single spaced) comments would be unusual and probably involve comments down to spelling issues. "A decent review" involves providing clear and constructive comments that will allow the editor to value the manuscript and the author to improve the manuscript.

So I would not say that a long review would necessarily be frowned upon, it clearly depends on how constructive it is. If someone spends a lot of effort improving language and grammar (which does not necessarily constitute the expectations from a review) that could be very useful. Normally such comments may be made as revisions in a file rather than a written report. So length is not a major issue, constructiveness is.

Answer 2

Speaking from the point of view of an editor: One of the best reviews I ever got was longer than the paper. The author, a young researcher, had proved three theorems, one of which I recognized as a known result. So I asked the original discoverer of that known result to referee the paper. In my cover letter, I mentioned that I recognized one of the theorems as his, and asked whether the other two theorems had enough novelty for a publication. It turned out that the other two theorems weren't new either. The referee could easily have just given citations for those two theorems and recommended rejection. Instead, he gave me (or, really, gave the author) a long, clear explanation of the state of the art in that subject, and he suggested some open problems that the author could try working on.

Answer 3

An "accept as is" option is useful after resubmissions; it signifies that no more work needs to be done. However, it is unusual to see that happen in an article on the first round of submissions. (I've had that happen precisely once in my career.)

Otherwise, I would say that the more detailed a review can be, and the more precise the suggestions for improving the paper are, the better it will be.

One to two pages is typically the norm; however, I have submitted a few three- to four-page reviews when I thought an article was already quite good, but could be better.

On the other hand, if a paper is already of relatively poor quality, I will explain the methodological or other significant flaws, but skip over an analysis of minute points; (it's simply not worth the time to rearrange the furniture when the roof is going to collapse any minute.)

Answer 4

As an author, reviewer, and key reader of a respected engineering journal, I can offer some perspective. The shortest review I received was one I solicited from a highly-respected professor at a prestigious university. His review was basically “This manuscript is not written well enough to be reviewed.”

The longest review I have received as an author was about five bulleted comments some of which were optional revisions and some minor but necessary clarifications; the shortest was one minor comment approving the manuscript.

As a reviewer I have on several occasions completely rewritten a non-English language author’s manuscript as a gratis professional service. To my surprise, I received thank you letters from the professional society publications chair and the editor thanking me for my ‘laudatory’ service. I took that to indicate my effort was unusual.

My shortest key reader review summary was to a VERY famous author who after a 22 page derivation, which he summarized as ‘simple’. With feigned seriousness, I ‘required him to remove the word ‘simple’ since he was on this uncustmary occasion communicating with mortals.