Anecdotal evidence: IEEE ICDM used to demand that all reviewers hold doctor titles, so there were no PhD students in the Program Committee. Then, in one particular year, all PC members were asked to review 17 papers (instead of the promised, and usual at ICDM, batch size of 8-10 papers), in a reviewing period of 16 days (including weekends). That did not lead to high-quality reviews either, and as additional fallout damage, several PC members declined the invitation for next year's ICDM, amplifying the problem.
I am not aware of any reasonable solution from the point of view of conference organizers:
- If we put criteria on reviewers, then fewer reviewers will be available, so the available reviewers must each do more work, which reduces their availability for other conferences.
- IJCAI this year asked their area chairs to rather aggressively desk reject papers: about half of all papers were filtered out before reviewing even begun. This decreases the load on PC members, but increases the load on area chairs, and it will inevitably increase the already unacceptably high variance inherent in the reviewing process: not all desk rejected papers will have deserved to be desk rejected, which is very unfair on those authors.
- ICLR does open reviewing, which means that reviews and reviewer identities are published on the internet. Since people know that their name will be available along with the review they wrote, only those people that know that they can invest a substantial amount of time will accept a reviewing invitation. This means that the reviewer pool will be smaller, which increases the load on individual PC members...
- If a conference were to monitor the performance of reviewers, and would somehow* manage to evaluate the quality of the reviews, then what? Suppose that PC members do a bad job. Do you kick them out? Then we have even fewer PC members to complete the job next year, increasing the load on the other PC members...
*how would we do this, though? Should the area chairs assess this? That would massively increase the workload of area chairs, which is probably not a good idea given the overload problems the rest of this post outlines.
As a scientist who tries to publish papers, I see two solutions on how to deal with the problem:
- Grow a thick skin when receiving conference reviews and trust that in the long run, the bias towards good papers will outweigh the variance in individual conference decisions.
- Submit your paper to a journal instead; if your papers gets an unjust bad review for a journal, you can at least discuss with the editor.