How do conferences / journals select reviewers?

Question

When you submit a paper / manuscript to a peer-reviewed venue, the organizers usually forward your work to 2-3 other "experts in your field" to assess your work.

I'm curious how this process of selection works, i.e. how is this pool of reviewers put together? and how are the best people from this pool chosen once there is a paper to review?

Is there an official list that each venue maintains and to which people can sign on if they want to be reviewers? (If so, I've never seen any "calls for reviewers" so how do they get people?)

Do organizers cold-call people/professors and ask them if they were willing to review a paper? If so, how are these people selected? [I realize that this might differ considerably, so I'm happy with anecdotal evidence, too]

Answer 1

In the journals I help edit (all in the medical fields), there are five ways we find reviewers for our manuscripts:

1. They have published with us before
2. They were nominated by the authors, editors, other reviewers or peers, or members of our editorial board
3. Our associate editors approached them through a scan of authors of papers on the same topic
4. They have volunteered to be placed on our list of reviewers
5. They work for us as consultants and are paid on a per-review basis (this is mostly relevant when we need specialist expertise such as in statistics or economics)

Once you are found, you are placed on a pool from which we can make selections on the basis of affiliation, expertise, seniority, etc.

As you can imagine, this is a rather valuable resource. We guard this list jealously and do not share this with other journals.

In my experience the response rate is typically 33%. That is, if we require five reviews, we will invite fifteen people. Thus, the pool needs to be deep in order to receive the requisite number of reviews for our purposes.

Answer 2

Examples from my experience:

• Anyone who has an account in the editorial management system for whatever reason. This could be former authors, former reviewers, former editorial board members, people who've written to us explicitly asking to be a reviewer, etc, as long as they have not indicated they are not willing to review or are not currently blacklisted (e.g. if a reviewer writes a very poor review, we might blacklist).
• Author-suggested reviewers (which we try to use sparingly since they often know the author personally)
• Reviewers suggested by other reviewers (e.g. if we invite, they decline but suggest X, we will invite X)
• From the references of the paper. If the paper is well-written then the introduction would give lots of related works, and the authors of these related works would be suitable to review the paper.
• Searching via Google Scholar, Web of Science, etc using keywords for the paper (sometimes but not necessarily the author-provided keywords).
• Personal contacts.

To choose people from the reviewer pool: choose the person with the closest-related expertise, as long as he/she is not already overburdened (e.g. already has an outstanding invitation, or has done three reviews in three months).

Cold-inviting reviewers does happen, and is a natural result from some of the methods mentioned above. It's not generally perceived as socially unacceptable however. If the reviewer doesn't want to review for the journal, they can decline indicating they're permanently not interested, we leave a note on their profile, and they're excluded from future searches.

Answer 3

Answering from the perspective of computer science conferences:

There is a committee, usually dubbed the "program committee". This is a group of of people that are invited by the conference organizers and this committee is formed in advance of papers being submitted. The size of the program committee varies from a dozen for a small workshop to thousands for big machine learning conference. The list is usually published and will change from year to year.

Sometimes the program committee does all of the reviewing. Papers are assigned by program chairs based on a bidding process. However, they may also bring in sub-reviewers. Sub-reviewers are third parties that are cold-called to review a paper because of their relevant expertise. Selection of sub-reviewers is up to the program committee members. Usually, they ask people they know, or if they are lost they look through the references of the submitted paper.