Theoretical computer science conferences usually have a review period of about two months. Programming committee members have potentially dozens of papers to review, alongside their regular day jobs. How thoroughly are papers usually reviewed? What kind of heuristics are often employed to review papers faster? Does correctness of every proof get verified in detail, or do reviewers put some trust into the authors getting it right?
-
1I just attended SoDA 2015, and although I didn't submit a paper, I believe that they only look at the abstracts. And if they want to know more about a paper, they will email the author asking for a full copy. How many papers are submitted and if they are reviewed as the full paper at all will depend on the conference.– Ryan DoughertyJan 13, 2015 at 18:18
-
There are lots of papers that fall in the category "even if all technical details are correct, this paper is not interesting enough for this conference". Helps a lot so that you can focus on the papers that should be accepted if they are correct.– Jukka SuomelaJan 17, 2015 at 0:52
-
This might be of interest to you: The NIPS experiment– Cape CodeJan 18, 2015 at 12:49
Add a comment
|
2 Answers
It varies. As a first year PhD student I was asked to review a theory paper, even though I had almost no background in the field. The professor explicitly said I only had to read the first half of the paper, and I only had to tell him if it looked interesting and correct.
At the same time I'm sure many reviewers do a diligent job and don't outsource their reviewing to inexperienced graduate students.
Related: http://www.phdcomics.com/comics/archive.php?comicid=1760
answered Jan 16, 2015 at 19:40
If a PC member has dozens of papers to review, then either a) the conference is doing it wrong or b) the PC member is expected to outsource most of the reviewing work to external reviewers. A good heuristic for not over-burdening the reviewers is to expect each reviewer to handle no more than 3-6 papers. With those sort of numbers, it's entirely reasonable to expect that each paper will get reviewed in great detail.
In practice, however, a great degree of variability in the quality of reviewers. Some reviewers will do a very thorough job---my theoretical papers often get back detailed comments from at least one reviewer that makes it clear that they worked through all of the math. Everybody in academia also has "bad reviewer" stories about people who didn't take their job seriously. For example, I recently saw a reviewer provide a review which, in its entirety, said, "Accept as talk." Good conferences with responsible program chairs try to mitigate these effects, but there's always a good deal of noise in the process.
answered Jan 13, 2015 at 16:51
-
7If a PC member has dozens of papers to review, then the conference is doing it wrong. — Theory conferences use a different PC architecture than other areas of computer science. For major theory conferences like STOC, FOCS, SODA, SOCG, etc., it is completely standard for each PC member to be assigned 30-40 papers. PC members are strongly encouraged to solicit external reviews for their papers, but those external reviewers are not part of the PC. It's not wrong; it's just different.– JeffEJan 13, 2015 at 16:59
-
4Also, theory conference PCs are kept small because PC members are forbidden from submitting to the conference, and because serving on the PC is a lot of work. As a result, serving on the same PC more than once per decade is rare, and serving two years in a row is almost unheard of.– JeffEJan 13, 2015 at 17:03
-
5@JeffE This sounds like a difference in terminology then... what these conferences are calling a PC member, I have more typically seen called a PC chair, and the external reviewers are equivalent to what I typically know as the PC members. I will update my answer accordingly...– jakebealJan 13, 2015 at 17:37