Is reviewing papers moving away from voluntary community service?

Question

I recently came across reviewing policies which make it a requirement for authors submitting a paper to a conference to also accept reviewing for it. Here is an example:

In order to submit paper(s) to EMNLP, you must nominate at least one author to serve as a reviewer, (usually the most senior author) and for that author to take on a full load (of up to 6 reviews).

I completely understand the rationale behind such decisions: the ratio submissions/reviewers cannot grow infinitely while maintaining a proper review process. Yet I'm concerned about the implications for the quality of the reviews, since people who are forced to do it might not be as careful as voluntary reviewers. This also feels like a step down from the traditional "community service" philosophy of the peer-review process.

Is this a general trend in the research community, i.e. is this happening in other domains as well?

• If yes, does this imply a change in the peer-review paradigm? For instance, as a researcher should I stop doing "free reviews" in order to keep my reviewing time for mandatory reviews?
• If no, are there other solutions being tested to the problem of ever-increasing papers/reviewers ratio?

Answer 1

I will address your three questions directly, and in-line:

"Is this a general trend in the research community, i.e. is this happening in other domains as well?"

I have never seen this before, and never heard of it. Everyone else that answered and commented, except for one person, has said that they have not seen it happen in their field either. The one person in the comments that did see the trend was Franck Dernoncourt, who says that it's definitely a trend in NLP (which probably means Natural Language Processing, not Neuro-Linguistic Programming). So, it is not a "general trend", but it is happening in some fields (such as yours and Franck's).

• "If yes, does this imply a change in the peer-review paradigm? For instance, as a researcher should I stop doing "free reviews" in order to keep my reviewing time for mandatory reviews?"

My answer is "no", but even if it were "yes", I do not see how that means you should "stop doing free reviews". The vast majority (by a long shot) of peer-review is still done for free.

• "If no, are there other solutions being tested to the problem of ever-increasing papers/reviewers ratio?"

I have never seen any other solutions tested. The peer-reviewed conferences I submit to, such as QIP (Quantum Information Processing) keep an extremely high standard for who is able to review. There is a committee of reviewers who are selected because they are recognized world leaders in their sub-fields, and each of them reviews several papers. Forcing any random person that submits a paper to review a paper, would lower the standards of the reviewers, and hence the standards of the conference.

Answer 2

More and more reviews are being done by graduate students. This holds for both conference papers and journals.

Sometimes this can actually lead to an increase in the quality of the reviewers, as many graduate students are immersed in the latest developments in their own particular field.

However, one of the problems with less experienced reviewers is that they often have a relatively more narrow range of expertise. They may be fine at reviewing papers in their area but might perhaps lack the breadth of experience to provide an informed judgement regarding work outside of their discipline or field of research narrowly understood.

This may seem off-topic but speaks to your broader point regarding the review of scholarly work being a part of the academic contribution to one's scholarly community.

There is already a division of labour where established academics will often taken on an editorial role and delegate reviewing tasks to more junior staff and graduate students. I think there is an increasing trend toward established academics no longer being the 'gatekeepers' of their discipline through paper reviews. This work is increasingly farmed out. I see the example you offer re conference presenters just being an extension of that.

It's a system. With lots of mythology around it. The esteem given to 'peer review' is one of those myths.

Answer 3

No, requiring authors to review is not a wide-spread practice or trend.