3

I have recently heard a few independent rumors about massive collusion amongst reviewers in major AI/ML conferences. The way this works is that a large group of program committee (PC) members start a chat group (on WhatsApp/Telegram etc.) and share their assigned papers; other members of the group flag papers that are theirs and get the "insiders" to score them higher.

To make things concrete, Alice and Bob start a chat where both share their assigned papers. Alice sees that Bob's batch contains one of her papers, and asks Bob to write a glowing review of that paper.

Are these rumors substantiated? Has anyone else heard of this going on? If it is as widespread as I was led to believe, what does this mean for the review process in these conferences and what countermeasures can we take?

|improve this question
  • 1
    Who sponsors these conferences? I would think that it would be treated as a serious breach in any conference sponsored by ACM or IEEE. – Buffy Feb 6 at 14:35
  • 1
    There is no question that this is a serious breach. AAAI, ICML, IJCAI and NIPS are not ACM or IEEE affiliated, but are widely considered as the top conferences in AI and ML. There is nothing to report if there's no concrete evidence of this. – Spark Feb 6 at 14:43
  • 7
    Asking if these rumors are substantiated without any evidence is simply spreading the rumors. – Brian Borchers Feb 6 at 16:14
  • @BrianBorchers fair point, but perhaps it's not a bad thing to have the rumors spread - it doesn't take a lot of people for this to work, especially if they are from the same sub-community within the discipline. – Spark Feb 6 at 16:21
  • 3
    @Spark Rumors spread for rumours sake is disinformation. Sometimes rumours emerge as a general feeling of distrust - but without indications or at least circumstantial evidence they are not of the falsifiable kind and thus at best a political instrument rather than an attempt to improve things. – Captain Emacs Feb 6 at 17:48
6

The top AI/ML conferences like NIPS have something like 2500 reviewers. There will probably be some bad apples in there who engage in the sort of behavior you describe. However, I would be shocked if this is happening at a large scale. The conference organizers would take action if they were aware of any such activities.

It sounds like you have no evidence of this. Frankly, it sounds like a typical conspiracy theory. Perhaps someone wishes to explain why their papers never get in but those from X, Y and Z always do, and decides that X, Y, and Z must be colluding. The true explanation will be a lot more mundane -- it's probably just that certain groups and people have more experience and have learned the "tricks of the trade" needed to publish papers.

|improve this answer
4

These are rumours and hearsay, as long as you do not have proof. If you have proof, it is a major scandal.

It is highly unethical to share papers sent to you for review unless you are permitted to delegate, in which case, that has to happen with discretion. Effectively, you are to treat a submitted paper as non-existing outside the purposes of the review/selection process (and I do not consider review trades/deals as legitimate).

I would not be surprised if there are some people who do that; if caught, they are likely to be officially blacklisted as reviewers. As far as to say this is a systemic approach, I am not aware of any evidence for this in strong conferences, in fact, all my personal experience points to rather the opposite. The reputation loss for a good conference would be irreversible. A single substantiated leak of such activity supported on a near-systemic level would be fatal and possibly irrecoverable.

Caveat: AI/ML has very recently experienced an explosive growth. It may be that, together with ensuing increased competitiveness, this has washed up some individuals of questionable integrity which instigate and/or take part in activities as described by OP. If that is the case, this is the time to act and nip these activities in the bud.

|improve this answer
  • The explosive growth in AI/ML submissions is exactly what makes me think that there is something to these rumors. – Spark Feb 6 at 16:19
  • 4
    @Spark While it is good practice to ensure that "Caesar's wife is above reproach", the possibility of something happening is not identical with something happening, and also not with suspicions of something happening. Proof is necessary in grave accusations like that. – Captain Emacs Feb 6 at 16:21
  • I'm not making any accusations of specific persons! If that wasn't clear, I apologize. I'm raising a concern that I heard from others, if I had proof I would have mentioned it. The AI/ML community is big and has a lot of subdivisions, perhaps this was happening for a while and I was being naive, perhaps this was already dealt with in other times - that's what I wanted to know. – Spark Feb 6 at 16:23
  • @Spark There is no guarantee that it does not happen. However, in the instances I was involved in in high-profile conferences, I was very pleased with which integrity things were handled. Of course things may have changed in the last few years, but even if they did, it may be an emerging process and so, whoever has knowledge of this should move fast to squash it before it becomes part of the system. Once engrained, it will be much harder to get rid of it. – Captain Emacs Feb 6 at 17:52

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.