8

My coauthors and I have submitted a paper for a well respected conference in a subfield of theoretical computer science. This is the kind of conference where the authors receive feedback before the final decision, and they have a chance to respond.

Some information:

  1. There were about 240 submissions. I know this because my paper was sent in the very last minutes and the submission server closed precisely at the announced deadline.
  2. The conference accepts between 25-30% of papers. In the last year there were about 80 accepted papers for about the same number of submissions.
  3. Each reviewer of a paper gives a score from the scale -2(strong reject), -1(weak reject), 0(boderline), 1(weak accept),2(strong accept).
  4. Each reviewer gives a confidence score to its review. I think that it varies from 1(not very sure) to 5(expert)
  5. Each paper has three reviews (ideally).

Once, when talking with a colleague that uses to participate on program committees he told me that in many conferences, a great percentual of papers are accepted/rejected only with basis on the score given by easychair, while those that are not so clear go to voting.

Questions:

  1. What percentual of papers are usually accepted automatically, i.e. only with basis on the automatic ranking system of easychair?
  2. Which score a paper should receive to be a clear accept? (assuming the paper has no flaws, and that no program committee member will claim to have found a flaw.)
  3. My paper got a score of (2,1,1), that is to say, one strong accept and two weak accepts. The reviewers confidence were not disclosed but from the reviews I could see that they were either 4 (high) or 5 (expert). Would such a score guarantee the paper to be on the safe side?
  • 1
    especially in TCS it's very hard to say. I had a submission with score (3,1,1) which was rejected, and another with (2,-1,-2) which was accepted (in different venues)! (The -2 was not because of any flaw, just profound disagreement on the model used). BTW many conferences have [-3,+3] range. – PsySp Mar 11 '17 at 12:47
  • Scores alone are meaningless. Suppose you get two accepts and one reject because one reviewer found a flaw in your result/proof/model/related work and the others missed it. Then you will probably be rejected despite having a good total score. Or the PC may read the reviews and decide that the strong scores are not justified (e.g. the reviewer didn't read it properly) or vice versa. – Thomas supports Monica Jul 11 '17 at 18:40
9

I've served on the program committees of several major theoretical computer science conferences, most recently SODA 2017.

What percent of papers are usually accepted automatically, i.e. only with basis on the automatic ranking system of easychair?

Negligible.

For every submission, the assigned reviewers are asked to come to an agreement whether to accept or reject. Each submission is assigned a shephard to guide the discussion and summarize it for the authors. For the very few (≤5) submissions that receive only strong-accept scores, this discussion is usually perfunctory ("This appears to be a clear accept; any objections?" "Nope."), but not always. The discussion is based on the content of the reviews, not just the scores (which may have come from subreviewers, who can't calibrate against other submissions).

For most conferences, the number of papers with supportive reviews is larger than the number of papers that can be accepted. Especially later in the decision process, PC members are often asked to rank remaining papers, including papers that they were not assigned to review. That instigates a higher-level discussion about this paper versus that one, or this AREA versus that one, which is rarely reported back to the authors (because the other submissions are none of their business).

In short: A paper with strong reviews can still be rejected because a majority of the other PC members preferred something else.

Which score a paper should receive to be a clear accept? (assuming the paper has no flaws, and that no program committee member will claim to have found a flaw.)

There is no such score.

I have seen submissions with scores of (2,2,1) rejected after discussion. Typically for such papers, PC members are encouraged to reduce their scores to reflect the discussion, but that does not always happen. I don't remember if I've ever seen a (2,2,2) submission being rejected after discussion, but that doesn't mean it can't happen.

My paper got a score of (2,1,1), that is to say, one strong accept and two weak accepts. The reviewers confidence were not disclosed but from the reviews I could see that they were either 4 (high) or 5 (expert). Would such a score guarantee the paper to be on the safe side?

Definitely not. Most papers with those scores are accepted, but definitely not all. Certainly I've had several conference rejections with similar scores.

More broadly: There is no "safe side".

  • what does it mean PC members are encouraged to reduce their scores to reflect the discussion? Do PC members reduce the scores given by sub-reviewers to better justify rejection? Does the opposite hold? – PsySp Mar 11 '17 at 12:15
  • When the initial reviews have wide variance, PC members sre encouraged to discuss the discrepancies and then edit iews and scores. For example, if a PC member initially rates a paper very highly, but another review points out a major bug, the first PC member might lower their score. Conversely, if one PC memebr thinks the problem is unimportant and initially gives a low score, enthusiasm from another reviewer might convince them to raise their score. – JeffE Mar 12 '17 at 1:42
  • Also, technically, the final score always comes from the PC member, not the sub-reviewer. If the PC member thinks the sub-reviewer is too generous, they might lower the subreviewer's score; if they think the sub-reviewer is too harsh, they might raise the sub-reviewer's score. – JeffE Mar 12 '17 at 1:44
  • That is very interesting. Then why the reviews sent to the authors have "Reviewers Confidence and Score" if they correspond to the PC member, even technically as you say? What's the point of the sub-reviewers if PC can change score and reviews at will? I think I will post it as a new Question because I had a rather (as I thought at that time) bizarre situation with the PC of a conference for which I was sub-reviewing. – PsySp Mar 12 '17 at 10:25
  • The point of sub-reviewers is to save PC members' time. PC members don't change reviews "at will", but in response to information unavailable to the sub-reviewer, in full view of other PC members. (EasyChair shows the complete revision history of every review.) Don't assume bad faith. – JeffE Mar 12 '17 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.