Why do some conferences have a variable number of reviewers for each submitted paper?
For example, I read on https://redd.it/9nomv1 that the AAAI 2019 conference has between 3 and 5 reviewers for each paper submission. Why isn't the number of reviewers the same for all submitted papers? Is that to make up for non-experienced or outside-the-area reviewers, or for reviewers who do not turn in reviews?
Editors usually invite more than the minimum amount of reviewers when they assign a paper for peer review, because not all invited reviewers will agree to do the review. A suggestion that I have seen in a guideline for editors (and which indeed mostly works well in my experience for this specific publication venue) is to invite 5 or 6 reviewers initially when aiming to get 2 reviews.
Occasionally an editor will be lucky and (almost) all invited reviewers will submit the review, leading to a higher than required number of reviews for this manuscript.
In other cases there will be less than the minimum number of reviews even though more reviewers have been invited, and the editor will have to invite even more reviewers.
The "number of reviewers" can mean three different things:
Those numbers can diverge:
Invitation Vs Acceptance (1. Vs 2.): The number of persons invited to review a paper and the number of people actually reviewing it can be different: sometimes, people decline, for many different reasons.
Acceptance Vs Production (2. Vs 3.) : The number of persons reviewing a paper and the number of reviews shared with the author(s) can be different: in some cases, a reviewer will turn out to be too late, or not knowledgeable enough, or produce a review the editors do not feel like sharing.
I know a conference, Journées Francophones des Langages Applicatifs, that has the following principle (translated by me):
Rejected paper have been reviewed by an additional person to ensure the correctness of the judgment: the rejected papers therefore receive at least 3 reviews.
The idea is that rejection needs to be well-motivated, so that the author(s) precisely understand what went wrong (clarity, relevance, technical incoherences, ...) and how they can improve.
That can also factor in the number of reviews actually shared.
But the bottom line, as silvado wrote, is that editors can decide how many persons they invite, not really how many reviews they will be able to share, even if having at least two decent reviews seems like the minimum.
