5

Most top conferences in Computer Science receive a record number of submissions every year, like AAAI received almost 9000 submissions this year, while CVPR 2020 received 6600 submissions. That would mean atleast 27000 and 20000 reviews respectively. That would mean almost 5000 and 4000 reviewers respectively, assuming 5 papers per reviewer.

I've also come across a few posts of area chairs (of these top-tier conferences) expressing their frustration of not getting reviews from the reviewers in time, and as a result there is a compromise on the quality of reviews. Is the burgeoning rise in ML/AI leading to falling standards? Is there an effective solution to this? Do such conferences have a way to check the quality of reviewers given that a lot of them are PhD students?

One criteria mentioned on their sites is that the reviewer should have 1-2 publications in such top-tier conferences, but is that enough? For instance, I have 3-4 publications (not in highly ranked conferences/journals), but I don't think I can provide effective reviews for papers submitted to those particular conferences/journals. Is there some additional criteria as well?

Improve this question
4
  • What are you asking? "Is there a solution"? "Do conferences check reviewers"? Or do you want to be a reviewer? Or "Are you good enough"?
    – Solar Mike
    Apr 21, 2020 at 13:49
  • I'm asking whether there is a compromise on the standard of reviews or not. If there is, because of the number, is there any solution to it? Do the conferences have a way to check the quality of reviewers? I gave my example to show my inadequacy, in reviewing papers where I have published work previously. I've edited the last part to make it more clear.
    – Jihadi
    Apr 21, 2020 at 14:39
  • One frame check: decreasing quality of reviews is not necessarily the same as decreasing quality of accepted papers. Also, both mean and variance are relevant here.
    – usul
    Apr 22, 2020 at 7:57
  • I think you mean top papers will get through in any case. That may be true, but a fair number of papers would be on the fringe (blog.mrtz.org/2014/12/15/the-nips-experiment.html), and could go either way depending on the reviews. Also promising papers which couldn't make the cut don't really get a chance to improve. Constructive rejections/reviews can go along way in improving quality of research.
    – Jihadi
    Apr 23, 2020 at 5:09

1 Answer 1

Reset to default
1

Anecdotal evidence: IEEE ICDM used to demand that all reviewers hold doctor titles, so there were no PhD students in the Program Committee. Then, in one particular year, all PC members were asked to review 17 papers (instead of the promised, and usual at ICDM, batch size of 8-10 papers), in a reviewing period of 16 days (including weekends). That did not lead to high-quality reviews either, and as additional fallout damage, several PC members declined the invitation for next year's ICDM, amplifying the problem.

I am not aware of any reasonable solution from the point of view of conference organizers:

  • If we put criteria on reviewers, then fewer reviewers will be available, so the available reviewers must each do more work, which reduces their availability for other conferences.
  • IJCAI this year asked their area chairs to rather aggressively desk reject papers: about half of all papers were filtered out before reviewing even begun. This decreases the load on PC members, but increases the load on area chairs, and it will inevitably increase the already unacceptably high variance inherent in the reviewing process: not all desk rejected papers will have deserved to be desk rejected, which is very unfair on those authors.
  • ICLR does open reviewing, which means that reviews and reviewer identities are published on the internet. Since people know that their name will be available along with the review they wrote, only those people that know that they can invest a substantial amount of time will accept a reviewing invitation. This means that the reviewer pool will be smaller, which increases the load on individual PC members...
  • If a conference were to monitor the performance of reviewers, and would somehow* manage to evaluate the quality of the reviews, then what? Suppose that PC members do a bad job. Do you kick them out? Then we have even fewer PC members to complete the job next year, increasing the load on the other PC members...

*how would we do this, though? Should the area chairs assess this? That would massively increase the workload of area chairs, which is probably not a good idea given the overload problems the rest of this post outlines.

As a scientist who tries to publish papers, I see two solutions on how to deal with the problem:

  1. Grow a thick skin when receiving conference reviews and trust that in the long run, the bias towards good papers will outweigh the variance in individual conference decisions.
  2. Submit your paper to a journal instead; if your papers gets an unjust bad review for a journal, you can at least discuss with the editor.
Improve this answer
4
  • Are reviewer identities public in ICLR? Aren't only author identities public? (E.g., a random example: I do not see reviewer names, or am I missing something?)
    – GoodDeeds
    Apr 21, 2020 at 18:22
  • Will putting a limit on the number of papers work? For instance, one person can be on the author list of 10 papers max. Would this be fair?
    – Jihadi
    Apr 22, 2020 at 5:13
  • @Jihadi Some conferences already do that. For example, IJCAI 2020 has put an upper limit of 6 submissions per author. This is down from 10 in IJCAI 2019. Even then they received 4700+ submissions.
    – GoodDeeds
    Apr 22, 2020 at 14:25
  • @GoodDeeds restricting it even further? But I guess top researchers may get the shot end of the stick. The way the system is going, one day its all going to crash on our heads.
    – Jihadi
    Apr 23, 2020 at 5:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .