I've received the reviewer reports for a paper I submitted to AAAI (the Association for the Advancement of Artificial Intelligence). I found their scoring method a bit confusing and I would like some help understanding their review of my work.

Each aspect is followed by a number and some plus and minus signs. For instance:


Can any experienced reviewers/authors explain the meaning of these signs and the range of scores?

  • 1
    Clarified slightly, including focus on AAAI (EasyChair allows many ranking systems) – jakebeal Oct 23 '14 at 14:58
  • So each reviewer gave a score and couple of signs for each aspect? I would guess 8(+++) means three reviewers evaluate this aspect to 8 – seteropere Oct 23 '14 at 18:12

I dug up my AAAI reviews from two years ago, and it looks like they've been tinkering with their scoring system, so I can't tell you a precise answer. I can give the general principle, however:

EasyChair allows the chairs of a conference to configure a set of "attributes" for scoring. Each attribute gets both a numerical value and a descriptor. Some examples from AAAI two years ago:

Individual attributes:

5: (ground-breaking (top 15%), a sufficient basis for accepting the paper)

4: (positive, a factor in accepting the paper)

2: (problematic, a factor in rejecting the paper)

Overall ratings:

2: (Strong acceptance. A 5 or 6 in some category, no 1 in any category.)

-2: (Strong rejection. A 1 in some category, no 5 or 6 in any category.)

So as you can see, then they were using a 1-6 scale and a -2 to 2 scale.

From the sample you gave, my guess is that they're currently using a 1-10 scale, with 5 being neutral. Thus

8 (+++)

Would translate approximately to "8 out of 10 (a very good score, three positive steps above neutral)."

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.