35

I recently received a request to subreview a paper for a top computer science conference. While it is related to my field of research, it is extremely long (the conference does not have a page limit) and complicated (it is an algorithm made of more than ten sub-algorithms, each of which is itself worth a conference paper). I find it very difficult to verify that it is correct. I mean, theoretically I may be able to understand it if I spend several whole weeks on reading every sub-algorithm, crafting small examples to myself, writing comupter simulations etc., but then I will have to stop doing anything else. What should I do?

  1. Decline the request, hoping that the paper will be given to a smarter reviewer that will be more capable of verifying its correctness? Or -

  2. Agree to the request, but tell the PC member in advance that I am going to review only a part of the paper which I can understand?

NOTE: The problem is not with the authors' presentation. In fact, their explanations seem good and clear. Just the algorithms themselves are highly complex.

Improve this question
7
  • 11
    From my experience, at least one or two of the reviewers don't know what the papers are about (that are mostly the "publish as is" reviewer comments...). That said, I (personally) would decline out of respect for the author and myself. If I can't really review a paper, I wouldn't.
    – Fl.pf.
    Commented Jul 21, 2017 at 5:41
  • 5
    @Fl.pf. I guess you are lucky (or we are not). It happens in about 50% of our submissions that a reviewer who clearly doesn't understand the research presented in our manuscript gives us a bad review with comments that make no sense whatsoever.
    – user64845
    Commented Jul 21, 2017 at 15:37
  • Is your subfield of computer science one in which conference papers are final publications?
    – Jeffrey Bosboom
    Commented Jul 21, 2017 at 17:53
  • 3
    While modesty is generally a good thing, I'd encourage you not to think of another reviewer as being "smarter" than you. We all have different areas of expertise.
    – Ellen Spertus
    Commented Jul 21, 2017 at 19:55
  • It seems to be rather common with reviewers who don't have many clues, but act as if they do. Probably discourages some percent of people wondering if they should go into academia each year. So I would go with 1)
    – mathreadler
    Commented Jul 21, 2017 at 23:28

10 Answers 10

Reset to default
31

I would go with 2, or at least contact the editor saying you would do 2, and asking if they'd prefer 1. There tends to be a shortage of referees, at least in my field, so I'd expect 2 to be preferred. I've been sent a paper to look at one aspect where someone else had apparently already checked a different aspect.

While I hate to say it, I believe in maths it would not really be expected that every tiny detail of a similar paper would be checked. I used to think it worked like that, but the time issues you mention are true for others too, so I guess 'certain enough' has to replace 'absolutely certain'.

Improve this answer
28

Accept it. In the worst case, your review could be something like

The paper is well structured and well written; while I was not able to confirm every single proof/algorithm, I am confident that the topics are presented in a sound way and would make an interesting topic in the conference.

This will make your position quite clear, and the publisher/organisator will easily be able to decide whether to take your review or not.

You will help to filter out badly written stuff, you will also be able to find flaws in the "internal" logic and the general approach, language, presentation etc.

Improve this answer
8

I would contact the editor or PC member who assigned me the paper, explain my limitations, and ask whether they would like to reassign the paper. They are in a better position to judge whether there are people better suited to review the paper or if you are the best reviewer available to them.

Another alternative would be suggesting a reviewer who is a better fit, if you know one and think they would be available.

Improve this answer
1
  • 3
    Why the downvote? If there's something wrong with my answer, I'd like to know what it is.
    – Ellen Spertus
    Commented Jul 21, 2017 at 22:37
6

By declining the review request for a conference (with sharp deadlines on notifications) you are not doing much favor to either the PC committee or the authors: the PC member invited you would have to spend more time searching for another replacement referee and this is highly non trivial and time consuming and, consequently, would give less time to the next referee for the report.

Note that probably the PC member that invited you is aware that you are not the expert. In many (most) cases, a relatively light review should be enough: nobody expect you (for a conference submission with a short deadline during summer) to read every little detail for the paper but rather to be able to understand and judge the results and the techniques. Are they relevant? Do they make good sense? Do they break some widely believe assumptions (sanity checks etc)?

It doesn't matter if you cannot verify every Lemma just make a high level recommendation based noting that as far as you have checked the results seem correct and reasonable. Of course you should mention everything to the PC committee.

Note that more than 50% of the reviews I have received of my submissions are 1 paragraph reviews: either saying great, super, accept and that's it, or criticizing that my paper is not relevant enough. In the beginning of my career I was slightly mad but I get to understand that expecting something more from all referees regarding conference submissions is unrealistic.

Improve this answer
6

As an editor of a journal, I'd encourage you to contact the editor stating pretty much what you've said here and offer to review the paper as best you can in light of those limitations, should they want that.

This is for two reasons.

  1. The editor is probably less of an expert in this than you are. Editors see all sorts of papers spanning dozens of subfields and do their best to find folks who have a reasonable chance of understanding what the paper's about and whether it's interesting. That you've been asked to review the paper means, somewhere, an editor thinks that you have a special knowledge of this particular subtopic.

  2. There is probably a lot you can do here, even if you don't have the time or ability to follow all of the authors' arguments. Does the argument appear well-crafted? Do the ideas fit together? Does the paper seem as though it will be of interest to people? Refer back to Point 1: from the Editor's perspective the paper may be jargony, rely on methods that are more obscure to them than to you, and may be of dubious interest. Your opinions may matter.

Improve this answer
3

Simply decline, and, if you really wish, say why and what you could do instead if asked. You are not up to the job to review the whole thing, and it's ok to declare so. If the paper is really astronomically complex, it would be the responsibility of the PC to choose an appropriate reviewing method, e.g., partition reviewing into subtasks and ask you (or others) to accomplish these subtasks.

PS. Personally, I would read and try to learn something new and useful from the paper. But it would be my own, subjective, personal decision, weakly related to your situation.

Improve this answer
2

If it was me, I would decline to review the paper. It doesn't seem fair to the author of the article to have an article half-reviewed and the most important part not even verified.

Furthermore, if it so happened that if the paper was published and someone else further down the line spotted problems in the paper then it would come back on the journal for allowing it to be published.

If you don't have the time or don't believe you can, then don't.

Improve this answer
1

Just give an honest review. Although it may not apply in your case, conference papers are often judged by different standards from archival journal papers, and a quick response that says, for example, interesting, should provoke worthwhile discussion (or whatever conference-like virtue the paper may have) will probably earn the organisers gratitude. Be honest about what you did not understand, or did not have time to understand. If you can analyse WHY you did not understand it, and suggest gaps that need to be filled, even more useful.

As others have pointed out, your imperfect review may be better than leaving the organisers searching, with limited time, for a more capable reviewer who may not even exist. If it is planned to publish proceedings, I would expect that another round of reviews will be organised, for which discussion following the paper may well be significant.

Improve this answer
1

I've encountered situations like this before, and if there is a limitation with the availability of potential new reviewers or time constraints, the solution I've found is to work with the panel of multiple reviewers. It's reasonable to request clarification on different points of the paper before proceeding with any final decisions, and normally the author is expected to have their draft work and notes readily available. But you should let the author know ASAP, as you should only evaluate the paper on the topic it discusses. Though you yourself may not understand or be able to review all of the paper, if you are confident in your credibility in your field, you should still give your input on whatever parts of it you do understand, and converse with other panel members on the parts that you don't, breaking the topics in the review process across multiple members if necessary. It's obviously not an accurate representation of the effectiveness of a paper if only one person with limited experience reviews it.

Improve this answer
0

You should entertain the possibility that the problem lies not solely with you, but with the authors' presentation. If you find it hard to understand, likely many others will struggle too. Have the confidence to state this plainly.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .