Inability to review a conference paper

Question

I am reviewing a paper for a conference. The paper contains a lot of work, and from a high-level perspective seems good.

But when I go into detail I can't actually parse the paper. I don't understand many things, even if the paper is within my field. I am a 2nd year PhD student that writes papers within the field. The paper is very close to my research topic.

Some of the causes that block me from correctly reviewing the paper might be:

• My impostor syndrome when seeing overly mathy papers.
• Their math formulation is not the traditional one and they do naming and formulation variations. Even though they can be properly defined.
• Over relies on the appendix to understand the content of the main body of the paper.
• The paper content is very dense.

I have spent already 3 days reviewing and I am not understanding it. The paper might be good/bad but I can't tell.

What review should I give? Push for acceptance, even if I can not tell if the paper is flawed/bad/good? Push for rejection when the problem might be that I don't understand it myself?

Answer 1

I do not agree with Kohan that a 2 years PhD student cannot review papers at least in my field (AI & ML). PhD students are often assigned by the PC members as sub-reviewers or directly as PC members by the SPC when they have papers in the same (or better) conference level.

Back to the question of the OP; as an author, I NEVER felt that the reviewers have understood my papers to a very high extent (especially when there is a detailed mathematical formulation) and as a reviewer, there are often disagreements between reviewers. In very few cases, I noticed reviewers trying to understand the math behind the papers they review and when they do, they are most probably PhD students because they are keen to read every word in the paper. In contrast, senior researchers have a lot of duties and usually a lot of papers to review so I do not believe they would spend one day per paper. Please note that I am not saying that senior researchers do not review papers seriously but they trust the authors in the mathematical formulation and their reviews are from a higher level perspective (you usually understand what is the role of the equation by understanding its context in the text).

As a reviewer, you are not supposed to understand everything (better if you do) but you need to write the reviews from your perspective and mention to the SPC that you have low confidence in your review. Note also that there is a deliberation in the end and you can adjust your review based on the discussion with the other reviewers.

If you are worried, you can write to the SPC and put him/her in the loop and follow his/her suggestion.

EDIT

After reading the comments, I think I have missed something in the OP's question. The OP mentioned that the paper was assigned to him without giving him the chance to accept or reject the invitation (As far as I understood from the comments). I find this an unusual situation (or the OP did not understand the invitation email) because the common practice in all conferences and journals I know is to send an invitation with the title and abstract, based on which the invitee assesses whether the paper is within his/her scope of expertise. If not, The SPC/PC/Editor expects the invitee to reject the invitation.

Answer 2

To convert my comments to an answer:

Return the paper to the editors and say that you have hard time reading the paper and suggest they ask a more senior mathematician to evaluate it. Personally, I would never ask a 2nd year PhD student to referee a paper. (However, in the past I did solicit referee reports from PhD students close to completion of their degrees.)

PS. Sometimes, the right answer to a question is "I do not know" or "I cannot do this." Knowing own limitations and being able to acknowledge these, is a sign of mathematical maturity. For all you know, by writing a lukewarm report, you might set up for rejection a great paper, written by an outsider to your research area, bringing nonstandard (for you area) tools and terminology.

Answer 3

As mentioned in the comments, odds are you are too junior to be reviewing this paper. However, you also mention that they're using nonstandard notation, which is not a good sign. You could return the paper to the editor and say you can't understand it, but there's an easy option: ask your supervisor. They're there to mentor you after all. Your supervisor can help identify if the paper is an obvious reject, and if not, they might be able to point you at resources to understand the paper. It's possible your supervisor will also tell you not to review the paper because you are too junior, but even in this scenario, asking your supervisor is much quicker than asking the editor, so it makes sense to do that first.

Answer 4

Publishing papers is not just about getting work out there, it is also about making other people understand the work done. As a 2nd year PhD, you are already part of the target audience of papers in your field.

As a reviewer, it is perfectly fine to come to the conclusion that a paper is not properly presented. Do not hide such a conclusion just because of the conclusion itself.

If your general impression is good, then a weak accept may be appropriate. If you are basically guessing the content, then a (weak) reject is no shame either.

---

• My impostor syndrome when seeing overly mathy papers.

This is really where you should put your focus on. Is the paper hard to understand because you generally struggle with mathy papers? Is the paper hard to understand compared to the usual mathy papers?

In a proper review, you can rate you degree of confidence. If you generally struggle with such papers, then a low confidence score is appropriate. This allows other, more thorough reviews to take precedence.

• Their math formulation is not the traditional one and they do naming and formulation variations. Even though they can be properly defined.

This seems like a clear shortcoming of the paper. Using obscure formulations – be it for math or prose – can hide veritable errors or blunders. At the very least, it makes it needlessly difficult to understand and/or reproduce the findings – one of the key goals of publications.

If you are feeling unsure about this point, check the paper's references. Do those use the same formulations? Is there a consistent formulation for key points that the reviewed paper deviates from?

• Over relies on the appendix to understand the content of the main body of the paper.

This depends strongly on the field and publication formats. In some it is perfectly fine to use the appendix for background and explanations that is already known to scientists in the field. In others an appendix is more of a technical addition to thoroughly reproduce a paper.

If you are unsure about this, note it as a comment to the editor. There may be formal rules about the appendix that have not been communicated to you.

• The paper content is very dense.

This ties in with some of the above points. Is the paper very dense compared to usual papers? Does the denseness make the paper needlessly hard to understand?

Try and identify specific points where content seems either superfluous or too short. It is fine for a review to suggest that some parts should be dropped in favour of expanding others. Take into account the appendix as well; it might be acceptable in the field to have a dense paper and supply lengthier explanations extra.

---

Answer 5

I would suggest giving the paper a general fairly positive review, or a lukewarm review, based solely on the high level introduction (if that's what you feel), while providing the lowest confidence degree (if that metric is available), and writing to the editor in the "note to editor (wouldn't be revealed to author)" that you understood the high level but not the details or technique and couldn't verify the proofs/arguments.

If you don't understand anything in the introduction, I would write that as far as you see you couldn't understand the merit of the paper from the current presentation (still specifying low confidence degree), and explaining that this is due to either inadequate exposition or your own misunderstanding (e.g., "I couldn't understand the overall merit...").

Answer 6

Well, I have a Master's Degree (not a PhD) and I've reviewed dozens of papers in my career, and I think I was more than sufficiently qualified to review them (even when some were written by PhD's). I don't believe the degree (or lack thereof) is the real problem -- a 2nd year PhD candidate should have more than enough academic knowledge to review technical papers. If the OP has not spent any time outside of school (i.e., from BS & MS directly to PhD), then it may be more a matter of lack of experience in the "real world" where they could have been exposed to more work being done in that discipline. Please don't take that as a criticism; it's more of a potential explanation of why they may be having some difficulty.

As far as reviewing papers is concerned, I use these general "rules":

• A paper to be presented at a conference or published in a journal is a way to present the results of some research or study to the world so others can benefit from that knowledge. The key is: COMMUNICATING the work done and the conclusions drawn from it. The subject (problem statement, proposed approach, assumptions, etc.) need to be communicated clearly enough for a practitioner in the field to understand, even when they aren't "experts". In this case, it sounds like that paper isn't doing well on this point.
• There should be enough explanation in the paper to allow a practitioner in the field to replicate the analysis described (although the data itself may not be accessible), and should include a sufficient level of detail on how that analysis was done. Obvious and common methods (like how to do ANOVA, for example) shouldn't be needed, but constantly referring to external references in the Appendix without providing specifics to more involved methods/tools does make it difficult for the reader to follow what the author is describing. It also doesn't allow the reviewer to determine if the author actually understands those methods and knows how to apply them correctly, specifically in the context of the paper.
• Also, if the author is not following standard notation or methods used by other practitioners in the field, then it undermines the 1st bullet (communicating clearly) as well.

There is nothing wrong with saying that major portions of a paper are unclear and confusing -- I've met many people who were clearly technical experts in their field, but were extremely poor communicators. That's made even worse if the author is writing a paper in English, when English is actually their second language. I know they can't help that themselves, but it is up to them to seek out a native English-speaking colleague to proofread and provide feedback before submitting it for a formal review. If that is the kind of review you're doing, then the feedback you provide is that much more important.

If it's really beyond your level of knowledge, there's nothing wrong with saying that you're not sufficiently qualified to do the review.

But DON'T just approve it to get it off your desk (there's far too many "junk" papers out there that should never have been published because some reviewers were a little lazy).

At the same time, rejecting something just because you don't understand it may deprive others from being able to benefit from that research.

That's just my opinion, but I hope my perspective helps you decide what to do.