Scope for the role of peer reviewer

Question

As a young scholar I frequently struggle with knowing the extent to which I should be critical of particular components of papers I am reviewing. I can formulate a rough hierarchy between major concerns (things that need to be changed or this shouldn't be published) and relatively minor concerns (things I think would improve the manuscript, but aren't substantive enough to affect whether the paper is publishable or not).The question here concerns the latter, in essence I don't know where the cut-off in reasonableness should be for minor concerns.

For one example, being curmudgeonly I would say bad graphics are the norm rather than the exception in my field, although they aren't frequently so bad I can't figure out what the author is trying to say. Are minor critques of graphs appropriate (e.g. your gridlines are very obtrusive, the aspect ratio of your chart isn't appropriate, your colors/patterns are hard to distinguish, you should use a dot plot instead of a stacked bar graph, etc.) Frequently my suggested improvements would be somewhat arbitrary though, so I frequently hesitate to give such advice.

Is there any advice to guide the role of the peer-reviewer? Another side question too, does the scope change if I'm doing this for a colleague versus as an anonymous reviewer for a journal?

Answer 1

I think everything you're asking about is within the scope of the reviewer's role. Generally speaking, I divide my review up into three sections:

Summary: A summary and free-form critique. Here I communicate what I think the "gist" of the paper is about, its strengths, and offer some weaknesses that might exist in the paper as a whole. For example, if I think the authors were slightly too timid in not offering a interpretation of the data, or if they've missed or glossed over some major point.

Major criticisms: These are things that must be fixed in order for me to consider it a publishable paper, and if I have the chance to review the revisions (some journals do this), things I expect to either see changed, or have very good arguments for why they're not. Stuff in this category includes:

• Flawed or inappropriate methods
• Major intuitive leaps that aren't supported by the data or analysis
• Study design problems that need attention
• Major failings of interpretation
• Journal specific problems, like failing to properly report your protocol to established standards for a clinical trial, or the inappropriate use of p-values in some journals.

Minor criticisms: These are all things that are essentially "The advice of someone reading your draft with a critical eye". Importantly, these are things where, if they all made it in, while I would possibly be annoyed, I wouldn't be upset that the paper hit the press. This does include things like advice on graphics (my pet peeve is graphics that are unintelligible when printed), missing citations, etc. Things that are above the level of a copy editor, but aren't going to move my decision on the paper one way or another, unless there are a lot of them.

Occasionally I'll put in one or two small copy editing notes if something jumps out at me (insure vs. assure vs. ensure, etc.)

The cut-off I use is "Will this irk me when I see it in print, and will I think less of the authors that produced it?" To use your example of bad graphs, yes, this would annoy me, and it results in a less usable finding than one with the appropriate graphs - just like garbled language in the Results section might. If its minor stuff, like a turn of phrase I wouldn't have used, or a slight fondness for run-on sentences? That falls below the radar.

As for friends vs. anonymous reviews, I think the scope does change slightly. For a friend, you're helping them polish a paper - I think a great many more things fall under that umbrella, including things like "That's really not what a semi-colon is for" or fiddling with the graphics parameters on a plot. For a reviewer, you're one of the last gatekeepers before this goes out into the world - but you aren't an editor. Your focus should be on the research, and the appropriate presentation of it, unless the errors are so bad as to impede one of those.

In either case, you should be polite.

Answer 2

Please see my answer to a similar question on the theoretical computer science SE site.

At @David's suggestion, I'm copy-pasting my answer here, but keep in mind that the question is slightly different, and my answer is aimed at theoretical computer science. The bold questions are quoted from the original post. Point 5 is probably the most relevant for this question.

Short version: Be respectful, but brutally honest.

---

1. What are the main criteria for determining the significance of a paper's results? To the best of your knowledge, does the paper make a significant, well-presented, and correct contribution to the state of the art? If the paper fails any of the three criteria, it's fair to reject it for that reason alone, regardless of the other two.

2. What are the main elements of a referee report, and which parts are most important? Here's what I think a report should contain. Everything should be visible to the author, except possibly for serious accusations of misconduct.

   a. A quick summary of the paper, to help the editor judge the quality of the results, and to help convince both the author and the editor that you actually read and understood the paper. Place the result in its larger context. Include a history of prior versions, even if the authors include it in the submission. Be respectful, but brutally honest.

   b. A discussion of the strengths and weaknesses of the paper, in terms of correctness, novelty, clarity, importance, generality, potential impact, elegance, technical depth, robustness, etc. If you suspect unethical behavior (plagiarism, parallel submission, cooked data), describe your suspicions. Be respectful, but brutally honest.

   c. A recommendation to the editor for further action — accept, accept with minor revision, ask for a second round of reviewing, or reject outright. Keep in mind that you are making a recommendation, not a decision; if you can't make up your mind, just say so. Be respectful, but brutally honest.

   d. More detailed feedback to the author — more detailed justification for your recommendation, requests for clarification in the final version, missing references, bugs in the proofs, simplifications, generalizations, typos, etc. Be respectful, but brutally honest.

3. How does assessment for conferences differ from that in journals? Conference reports should be shorter; program committees have hundreds of papers to consider at once. Whether there should be a difference between conference and journal papers is up to the journal (and indirectly, up to the community). Most theoretical computer science journals do not insist on a significant difference; it is quite common for the conference and journal versions of a theory paper to be essentially identical. When in doubt, ask the editor!

4. What if I don't understand the paper? ...the proof? (Is it my fault or theirs?) If you still don't understand the paper after making a good-faith effort, it's the author's fault, or possibly the editor's, but certainly not yours. The author's primary responsibility is to effectively communicate their result to their audience, and a good editor will send you a paper to referee only if they think you're a good representative of the paper's intended audience. But you do have to make a good-faith effort; do not expect to immediately understand everything (anything?) immediately on your first reading.

5. What about typographical/grammatical mistakes? If there are a lot of errors, don't even read the paper; just recommend rejection on the grounds that the paper is not professionally written. Otherwise, if you really want to be thorough, include a representative list of grammar, spelling, and punctuation mistakes, but don't knock yourself out finding every last bug. Be respectful, but brutally honest.

6. How much time should I spend on a report? Expect to spend about an hour per page, mostly on internalizing the paper's results and techniques. Be pleasantly surprised when it doesn't actually take that long. (If it takes significantly less time than that, either the paper is either exceedingly elegant and well-written, you know the area extremely well, or the paper is technically shallow. Don't confuse these three possibilities.)

7. How many reports a year am I expected to write? You should write at least as many referee reports as other people write for you. If this takes more time than writing your own papers, you're not spending enough time on your own papers.

Answer 3

PLoS Computational Biology editor P.E. Bourne has written a series of very decent “Ten simples rules to …” articles. There is no detailed guide to being a good reviewer, but Ten Simple Rules for Reviewers sure is a good starting point. Summing it up:

1. Do Not Accept a Review Assignment unless You Can Accomplish the Task in the Requested Timeframe—Learn to Say No

2. Avoid Conflict of Interest

3. Write Reviews You Would Be Satisfied with as an Author

4. As a Reviewer You Are Part of the Authoring Process

5. Be Sure to Enjoy and to Learn from the Reviewing Process

6. Develop a Method of Reviewing That Works for You

7. Spend Your Precious Time on Papers Worthy of a Good Review

8. Maintain the Anonymity of the Review Process if the Journal Requires It

9. Write Clearly, Succinctly, and in a Neutral Tone, but Be Decisive

10. Make Use of the “Comments to Editors”

I would add: carefully read both the “instructions for authors” and “instructions for reviewers” of the journal you are reviewing for, if you are not already very familiar with them.

Answer 4

There are certain things I look for when reviewing papers that are "show stoppers", i.e. if I find one of these, I usually recommend rejection.

• Things that don't add up, e.g. fractions that don't add up to 1, unreasonable large performance improvements such as orders of magnitude when the proposed method would save 50% of the work in the best case.
• Major omissions, i.e. something that would be required to reproduce the results/implement the method is not described adequately.
• Wrong or unjustified statements.

The things that don't fall into this category (e.g. graphs that are hard to read) are usually not a reason for me to recommend rejection, unless they occur several times throughout the paper. The same goes for bad spelling/grammar.

I would always point out things like you mention as example (one particular thing I don't like is graphs with different scales next to each other to compare two approaches) if you think that changing them would make the paper better. In the end, everything is subjective and may seem arbitrary.

I would use the same diligence regardless of whether looking something over for a colleague or reviewing for conference/journal because even in the former case the paper will presumably be submitted somewhere where it will be peer-reviewed.

In summary, I think that there need to be a number of minor concerns throughout the paper to recommend its rejection. One or two graphs that look odd would for me not be a reason to do so, unless of course there are other, more serious problems.

Answer 5

Here is the rough checklist I use for refereeing. I work in mathematics, so it may need adjustment for other areas. When I write a review, I imagine it is for my own paper, and I try to include the things that I would like a reviewer to include for me.

1. Is the paper correct? Trivial errors can be corrected in a resubmission, of course. I have not yet received a paper in which I found any serious error.

   1. Part of correctness is clarity. I consider the average readership of the journal (to the extent I can), and any aspects of the paper that would be unclear to them can be raised in the report. There is a difference, of course, between unclarity and writing style. But papers should use standard terminology when it exists, and proofs should be written in a way that is not unduly difficult to follow. Occasionally, a referee can suggest a more streamlined proof, which is fine.

   2. I keep a running list of typos, style errors, and other trivialities, which I list at the bottom of the report with minimal commentary.

   3. I do not check the correctness of all bibliographic data, but I often refer to one or more of the references when reading the paper. If I notice any errors in the bibliography I note them in the report.

2. Is the paper complete?

   1. Are there any obvious gaps in the research? For example, if a theorem has a strange additional hypothesis, the author should address the necessity of this hypothesis, or consider stating it as a question. Nobody else will be able to publish a paper to fill in minor gaps, so it is important for the author to be sufficiently thorough in the original paper, for the sake of the overall literature.

   2. Are there additional references that should be cited? If I know of additional research that the author has not mentioned, I can raise it in the report. I believe this is one of the more important roles of the reviewer, because no author is aware of the entire research literature.

3. Does the paper fit the journal to which it has been submitted? I have a general feeling for the differences between the journals in my area. Most journals have a statement of scope and purpose on their website, as well. For some journals, I think that all professional-quality, on-topic papers are in the scope of the journal. But for "high-tier" journals, the paper needs to have sufficient results (and, perhaps, a sufficient density of results) to fit.

   1. The degree of completeness (item 2) can be important here. The editor will make the final decision, of course, but it is not inappropriate for the reviewer to indicate if they think the paper is correct but not a good fit for the journal to which it is submitted.

   2. If it is clear that a paper is not a good fit, I may write a report indicating this without verifying the correctness of the results. This allows the author to resubmit to another journal more quickly than if I took my time with the paper.

4. Did the editor give any additional instructions? For example, one top-tier journal explicitly asked reviewers to raise the quality level required for acceptance, because the journal had a long backlog and wanted to reduce the volume of accepted papers. Other uncommon situations can arise, which the editor may summarize for the reviewer.

I attempt to answer questions 1 and 3 in the first paragraph of the review. These are what the editor needs to know. The lower parts of the review are intended more for the author, and may have remarks related to possible changes or additions, particularly if my recommendation is to revise and resubmit.

Answer 6

If you decide that a particular paper you're reviewing is acceptable, then your job as a referee of the paper becomes to suggest improvements that you think will substantively help the presentation of the scientific content. Even minor corrections like the ones you've listed can be considered substantive when they lead to "real" improvement.

For example, I'd argue that suggesting a comment that axes labels should be 12 pt instead of 7 pt would be appropriate, as it would substantially improve legibility; however, arguing for 12 pt instead of 13 pt is not significant enough to make a real difference. Similarly, I would point out grammatical or spelling mistakes if they are relatively few in number, but make a general comment if there are many, because I'm not being paid to be a copy editor. (As another example, Oxford commas wouldn't rise to that level, unless it's an "Eats, shoots, and leaves" issue.)

In general, however, there isn't anything that is really too nitpicking in nature unless all you're doing is expressing a personal preference.

Answer 7

If these are suggestions rather than mandates (as you suggest) and if you are willing to commit the time to providing such feedback, I'd say go for it. It could only help, as long as you're not so picky that the authors end up with a list that is too overwhelming to deal with (not sure where the bar is for that).

FYI: I'm a "young" scholar, too, not a journal editor.

Answer 8

I can also add some other pointer I've come to learn over the years on what constitutes a good reviews.

Things you should avoid:

• Rejecting the paper because you did not like the approach (science is not about liking, is about correctness)

• Rejecting the paper because it has many typos (you can always ask for a spell checking, but if the idea is good, not being a native speaker shouldn't affect you)

• Criticizing a paper for not having simple definitions, my rule of thumb is that if a concept pops in a Google search as a wikipedia page, the author does not have to explain it again.

Things that can help the paper:

• If the Grammar is poor, recommend some book for writing styles, like Strunk and White, and give some examples on how the paper can be improved.
• If you think the paper has some obscure concepts (that did not come up in Wikipedia) ask for clarification.
• The paper, as a rule, should be implementable by someone who has the expertise in the area.

Answer 9

My main criterion for such concerns over clarity and presentation is reader time: is the time it takes the authors to address your concern comparable to the (cumulative) time gained by the paper's readers.

This also means: the larger the expected audience, the more you can nitpick.