9

In the papers I've submitted to ACM conferences, in the reviews there is typically a field next to the reviewer that appears to give the reviewer's level of proficiency in the topic. For example, it might say "expert".

How is that established? What significance does this rating play? When a level of proficiency is part of the review form, is it possible to leave this blank or unranked?

1
  • I have been involved in several PCs where the absence of a (self-declared) expert reviewer was seen as a source of concern, and additional, expert reviewers were added to certain papers. Dec 18, 2023 at 11:36

2 Answers 2

7

To extend Maeher's great answer: ACM does not impose a specific way to run a conference. Standard conference software allows the Programming Committee (PC) chair to allow self-identification of reviewers regarding their confidence in their own review. These self-identifications are open to the members of the PC. I remember one instance when we received a complaint about a rejection because the "expert" had advised acceptance and the "familiar with the field" had not. Usually, in a case like this, the PC discusses, and in this case, the majority of the PC felt that the expert should be rated "familiar with the field" and the familiar with the field reviewer should be considered the expert.

Now, some conferences do not have a PC meeting to finalize the program. Most people invited to be on a PC (in my experience) do not feel that averaging even with weighting according to expertise of reviewers is a good way.

However, the review grades alone are used to "weed out" bad or ill-fitting papers and admit papers with unanimous reviewer recommendations because PC members cannot look at all papers. At a smaller conference, sometimes a paper is read by a single, a couple, or a few PC members while the committee discusses other papers. They then come back with a verdict on how to evaluate the reviews and the paper. If you receive a very terse review from an "expert", this might have happened in your case. That is because a PC often feels that leaving an author without feed-back is worse than providing a feed-back that the reviewer had to form within half an hour and based on the already submitted reviews.

4
  • Hmm... if the rating is used, if at all, by the program committee, then I can't see any benefit in reporting this information back to the submitter. The harm is of course giving a false impression of certainty when it might not be deserved. Furthermore, if this kind of thing happens, it could promote the impression of academics being arrogant. In my experience with dealing with people in general, most people will err on the side of over inflating their judgement and abilities than underestimate them.
    – rocky
    Dec 18, 2023 at 1:54
  • 3
    I think that, from a program chair's perspective, the rating is most useful when a reviewer claims not to be an expert. Many reviews by non-experts miss big issues and are overly positive, or they don't understand why the problem being addressed by the paper is important, and are overly negative. In those cases, program committees know to not weigh these reviews very highly. It is useful to authors because they, too, can see why a specific review was not considered as important as some others. Dec 18, 2023 at 4:09
  • "they, too, can see why a specific review was not considered as important as some others." Hmm... I wonder then if the individual ranking isn't that helpful over just the aggregate after it has gone through the complex weighting with hidden parameters. But that's just me where I am interested in actionable feedback and end result.
    – rocky
    Dec 18, 2023 at 5:10
  • 1
    "I think that, from a program chair's perspective, the rating is most useful when a reviewer claims not to be an expert." The example given by Schwarz is specifically the opposite kind of situation: the majority of the PC felt that there was a problem with overinflation of one individual along with underinflation of another.
    – rocky
    Dec 18, 2023 at 7:01
15

While each conference can organize the reviewing procedures pretty much however they want, that rating is usually simply self assigned. I.e., the reviewer gets to choose what they consider their expertise to be.

This information is at least used by the other members of the program committee (in particular the chairs) to weigh whatever the review says, in particular when reviews disagree. At least in some conferences it is also used as a formal weight for averaging the scores assigned by the different reviewers.

6
  • 2
    I will add that while the rating is self-assigned, there is generally no real incentive for the referee to claim to be an "expert" on a topic when they are not. (Of course, their self-perception might be skewed, but that's a different matter.) Dec 17, 2023 at 21:11
  • 1
    @AdamPřenosil There's no real incentive, no. But my long-term observation is still that these expertise scores are notoriously inflated, to the point of being basically meaningless.
    – xLeitix
    Dec 18, 2023 at 6:07
  • 1
    @AdamPřenosil Two incentives are (although only marginal, but can accumulate when done often): 1. Greater perceived feeling of influence, which is a psychological incentive. 2. Self-promotion - especially for a fashionable topic, people might like to be perceived as experts, to be invited for relevant collaborations and roles. Dec 18, 2023 at 11:32
  • @lighthousekeeper Interesting - both of these incentives are of no value to the person submitting a paper since the reviewer is anonymous.
    – rocky
    Dec 18, 2023 at 12:46
  • @rocky An indirect value might be that a person who self-declares as expert might be more interested to write a decent review, since they cannot claim lack of expertise as an excuse for writing a superficial review. Dec 18, 2023 at 13:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .