As a reviewer, you have a dual role: you are both, a gatekeeper responsible for ensuring the quality of the programme, and a mentor to the authors for improving their own work.
- As a gatekeeper, your task is to fairly evaluate whether a paper fulfills the standards of the community and this specific conference, whether the paper is methodologically sound, and (notoriously) whether the paper is sufficiently interesting to the community. The last part tends to be the most subjective, and my personal recommendation here is to not put too much stock on this item. By and large, we as a scientific community tend to overestimate our ability to predict what research will have an impact. Further, don't be too strict with regards to methodological soundness - my impression is that particularly inexperienced researchers have a tendency to be overly critical of other people's work. Finally, be careful not to recommend rejecting a paper simply because it is not the kind of research you would have done, or because the authors have chosen a different method than the one that you would use when attacking their problem.
- As a mentor, you want your review to be both, honest and helpful. Don't sugarcoat your review, but make sure to highlight positive as well as negative aspects of the manuscript. If you criticise, do so with concrete, actionable recommendations. Don't just write that "approach A is bad", but suggest what else the authors could do that would be more convincing. Don't write that the authors have not taken important related work into account, but list what exactly is missing and why this matters. Always, always make sure that your review is professional and friendly - you may be tempted to give some authors some "tough love" and set them straight. Avoid this temptation at all costs.
Some more recommendations:
- Don't write a super-short, high-level review. As you are surely aware, as an author it is very disappointing to receive a 5-lines review for an article that you spent a lot of time on. However, it is ok if the length and detail corresponds to the quality of the paper - for a very questionable submission that is clearly below the bar, you don't need to write pages and pages of explanations if one or two paragraphs are sufficient. For instance, if you are convinced that the method the authors use is fundamentally incorrect, there is no point in detailedly commenting the entire manuscript.
- Make sure that the crucial points of your review stand out. If you write a detailed review (as you should), highlight what the most important comments are, so that they are not drowned out by smaller, detailed comments. If you recommend rejecting a paper, I suggest clearly differentiating issues led to you rejecting the paper and things that you think are either easy to fix or less crucial problems.
- Give the authors the benefit of doubt. Don't assume that everything that isn't described in sufficient detail is necessarily terrible, or that every mistake in the paper has been planted with malicious intend.
- Relatedly, make sure that you don't read anything into the paper that isn't actually described. Our mind tends to fill in the blanks, and sometimes all it takes is a few keywords to fill your mind with an idea of what you think the authors have done. Make sure that you are evaluating what the authors actually describe, and not what your mind imagined after the abstract.
- You have a duty to try to understand the work, but if you fail to understand what the authors actually did even after honestly trying to follow, it is on the authors. Ultimately, you are a prototype reader for the paper - if you fail to follow the manuscript despite presumably investing more time and goodwill than the average reader, the paper may need a better presentation.
Overall, remember your own experiences as an author - which reviews did you find helpful? Write reviews that you yourself would appreciate, especially if you recommend rejecting the paper.
