1

Suppose you are submitting a paper to a conference, your draft paper has been submitted, and it will take approximately four month in order for the reviews to come back, the acceptance/rejection will be given and if accepted you will be told to submit the final draft.

Can an insider shed some lights into what exactly happens in between these four months for a typical conference in technical fields such as computer science or engineering?

For instance, how many people are usually on the peer-review committee? Are these usually professors or PhD students? How do reviewers inform each other about their opinions and comments? How long does it actually take (during the span of the review period) to evaluate a single paper? Finally, how are decisions made by the reviewers evaluated at the end? What if there is the same number of people in favor of rejecting a paper versus accepting one?

6

This is my experience in software engineering, a subfield of computer science.

For instance, how many people are usually on the peer-review committee?

This committee is typically called program committee (PC), and the number of involved members differs significantly between conferences - I've seen numbers between 25 and more than one hundred. The PC is usually listed on the website of the conference.

The number of reviewers for any single given paper is usually 3 or 4, and each PC member may be assigned to review a larger number of papers, often between 10 to 25. To manage this workload, PC members may delegate their reviews to colleagues or PhD students. Consequently, the actual number of reviewers involved in the conference can be much greater than the PC size indicates.

In addition, especially at larger conference, there can be second tier of reviewers, called "Expert Review Panel" or something similar, whose members typically review a smaller number of papers. For example, they can be assigned to papers if they are at the borderline between acceptance and rejection after the initial review.

Are these usually professors or PhD students?

Mostly professors, some post-docs, and, in some rare cases, late PhD students. However, again it's worth pointing out that professors may delegate some of their reviews to their PhD students.

How do reviewers inform each other about their opinions and comments?

To organize the reviewing process, conferences typically use a dedicated software called conference management system, such as EasyChair.

If you have submitted your paper using some website, it was probably a conference management system - apart from the author perspective, there's also a dedicated perspective for PC members.

How long does it actually take (during the span of the review period) to evaluate a single paper?

Reviewers are typically given between one and two months. Some conference have a rebuttal system, where there are actually two rounds of reviews, with an author response period in between. This would then be specified in the call for papers.

Finally, how are decisions made by the reviewers evaluated at the end?

There are different options: There can be a physical PC meeting where all PC members discuss the papers in the same room, a virtual PC meeting using some online platform, or even no PC meeting at all, but just a discussion period in the conference management system.

In any case, there is a discussion about each paper, and the discussion needs to end with an unanimous agreement between the PC members. Since the discussion is open to all PC members, it can also happen that people contribute to the discussion of a paper that were not originally assigned as reviewers.

What if there is the same number of people in favor of rejecting a paper versus accepting one?

It all depends on the discussion. It happens that the numerical decision on a paper is actually reversed based on the discussion.

  • 1
    This answer is quite good, but I think it could be somewhat misleading concerning the fact that the number of reviewers for any single given paper is usually somewhere around 3 to 5, and that these can very commonly indeed be PhD students. If I imagine reading this answer without knowing anything about the procedures, I could easily conclude that a submitted manuscript is thoroughly analyzed by more than 25 professors. – O. R. Mapper Mar 24 '17 at 8:54
  • +1 The only thing I would add here is that each PC member usually is responsible for reviewing a large number of papers in these 1-2 months, often between 10 and 25. Consequently, one does not have a month time to review a paper, one has a month of time to review or delegate up to 25 reviews, in addition to one's usual workload. That is, it's usually infeasible to invest more than a few hours or a workday on a single conference review. – xLeitix Mar 24 '17 at 8:59
  • Thanks, both comments are right, and I updated the answer accordingly. – lighthouse keeper Mar 24 '17 at 9:05
  • @xLeitix 10-25 papers seems very light. For major theory conferences, each PC member is responsible for 30-50 papers. Theory PCs are kept deliberately small, because PC members are forbidden to submit anything themselves. – JeffE Mar 24 '17 at 13:38
  • @JeffE The things you learn. If reviewers at ICSE etc. would have to review 30+ papers and be prohibited from submitting themselves, the conference may find itself without a PC. – xLeitix Mar 24 '17 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.