To what extent do Easychair Scores Affect Acceptance/Rejection in Theoretical Computer Science Conferences?

Question

My coauthors and I have submitted a paper for a well respected conference in a subfield of theoretical computer science. This is the kind of conference where the authors receive feedback before the final decision, and they have a chance to respond.

Some information:

1. There were about 240 submissions. I know this because my paper was sent in the very last minutes and the submission server closed precisely at the announced deadline.
2. The conference accepts between 25-30% of papers. In the last year there were about 80 accepted papers for about the same number of submissions.
3. Each reviewer of a paper gives a score from the scale -2(strong reject), -1(weak reject), 0(boderline), 1(weak accept),2(strong accept).
4. Each reviewer gives a confidence score to its review. I think that it varies from 1(not very sure) to 5(expert)
5. Each paper has three reviews (ideally).

Once, when talking with a colleague that uses to participate on program committees he told me that in many conferences, a great percentual of papers are accepted/rejected only with basis on the score given by easychair, while those that are not so clear go to voting.

Questions:

1. What percentual of papers are usually accepted automatically, i.e. only with basis on the automatic ranking system of easychair?
2. Which score a paper should receive to be a clear accept? (assuming the paper has no flaws, and that no program committee member will claim to have found a flaw.)
3. My paper got a score of (2,1,1), that is to say, one strong accept and two weak accepts. The reviewers confidence were not disclosed but from the reviews I could see that they were either 4 (high) or 5 (expert). Would such a score guarantee the paper to be on the safe side?

Answer 1

I've served on the program committees of several major theoretical computer science conferences, most recently SODA 2017.

What percent of papers are usually accepted automatically, i.e. only with basis on the automatic ranking system of easychair?

Negligible.

For every submission, the assigned reviewers are asked to come to an agreement whether to accept or reject. Each submission is assigned a shephard to guide the discussion and summarize it for the authors. For the very few (≤5) submissions that receive only strong-accept scores, this discussion is usually perfunctory ("This appears to be a clear accept; any objections?" "Nope."), but not always. The discussion is based on the content of the reviews, not just the scores (which may have come from subreviewers, who can't calibrate against other submissions).

For most conferences, the number of papers with supportive reviews is larger than the number of papers that can be accepted. Especially later in the decision process, PC members are often asked to rank remaining papers, including papers that they were not assigned to review. That instigates a higher-level discussion about this paper versus that one, or this AREA versus that one, which is rarely reported back to the authors (because the other submissions are none of their business).

In short: A paper with strong reviews can still be rejected because a majority of the other PC members preferred something else.

Which score a paper should receive to be a clear accept? (assuming the paper has no flaws, and that no program committee member will claim to have found a flaw.)

There is no such score.

I have seen submissions with scores of (2,2,1) rejected after discussion. Typically for such papers, PC members are encouraged to reduce their scores to reflect the discussion, but that does not always happen. I don't remember if I've ever seen a (2,2,2) submission being rejected after discussion, but that doesn't mean it can't happen.

My paper got a score of (2,1,1), that is to say, one strong accept and two weak accepts. The reviewers confidence were not disclosed but from the reviews I could see that they were either 4 (high) or 5 (expert). Would such a score guarantee the paper to be on the safe side?

Definitely not. Most papers with those scores are accepted, but definitely not all. Certainly I've had several conference rejections with similar scores.

More broadly: There is no "safe side".