Should researchers/journal editors ever solicit referee reports from companies that are negatively affected by the research?

Question

This is a general question inspired by an interesting question where a researcher investigated the efficacy of a product produced by a company and made adverse findings against the product. In that question, the company became aware of the research prior to publication and some of its personnel sought input into the research (e.g., requesting the data, requesting to see the paper prior to publication), and the researcher wanted to know how to deal with this.

There are some obvious dangers that arise when personnel in such a company seek input into the research process for an outside review of one of the company's products, since it could potentially break the independence of the research. However, there are also a number of potential benefits: (1) company personnel may be able to share expert knowledge of the subject, perhaps provide useful information, interpretation, criticism, or data; (2) the researchers potentially benefit from critical feedback on their work (from a potentially hostile reviewer) prior to publication, allowing them to strengthen the work with revisions; and (3) the review process could potentially avoid ---or reduce the scope of--- possible false research findings and resulting legal claims. In view of the possible benefits of this kind of review, I have a number of questions (feel free to respond to only one/some of them):

• Should researchers or academic journal editors ever seek referee reports (or more informal feedback) from personnel from companies who are adversely affected by outside research of their products/services/etc., prior to publication of results? If so, what circumstances would make this appropriate?

• If such referee reports are solicited (and given) as part of the formal peer review process, how should an editor treat them? Should they be treated just like any other referee reports, or should the editor take them "with a grain of salt"?

• If such referee reports are solicited (and given), either in formal peer-review or in informal communication with the researchers, what if any disclosure of this should be made in the paper? Should the report just be treated like any other referee report and remain anonymous, or should the reader be made aware of any changes made to the paper in response to referee review by the company?

• Is there any clear current practice for this in universities or academic journals? Have there been any cases that have caused controversies or misconduct issues? Is there a "best practice" on this issue?

• Is this all just a complete no-no (i.e., reports should not be solicited from interested companies)?

Answer 1

First, companies do not review papers. People review them, and as an editor you need to ensure that you know who actually wrote the review.

Second, the rule in general is: People who have a conflict of interest should generally be avoided, and if you need to choose them (for example, because there simply are not enough other people in a field who are not conflicted), they need to disclose the conflict of interest and when you make a decision about a paper based on a conflicted reviewer, you need to take the conflict into account.

For this particular case:

• If that is possible, as an editor I would try to avoid using a reviewer from the company simply because they have real or perceived conflicts of interest. (Note that a perceived COI is just as bad as a real COI.)

• If it is not possible to avoid using someone from that company, or because you want an insider opinion, you can do so, but you have to consider how much weight you want to give that review when making a decision. It also seems appropriate to state in the reply to the authors that Reviewer #4 works at Company X (without disclosing that person's identity), and that you have taken this fact into account when making your decision. Personally, if I had to do this, I would probably write a longish description of which of the points reviewer #4 made I consider valid, and which ones I think are points that I believe are tainted by perceived COIs and that can consequently be ignored by the authors (but can of course still be addressed if they so wish). I would also lay out this kind of procedure when asking the person from Company X, to make sure that they understand how I will treat the review I ask them to provide.

In the end, the goal is to ensure that you get good feedback on a paper and to have a (documented) procedure to guarantee impartiality.

Answer 2

This reminds me of the AlphaZero paper. AlphaZero is a game-playing AI by DeepMind that learned to play chess. Naturally DeepMind compared AlphaZero against Stockfish, the strongest conventional chess engine at the time. However their inexperience with Stockfish showed, and their tests had serious problems. When they made their preprint public, a Stockfish developer wrote:

The match results by themselves are not particularly meaningful because of the rather strange choice of time controls and Stockfish parameter settings: The games were played at a fixed time of 1 minute/move, which means that Stockfish has no use of its time management heuristics (lot of effort has been put into making Stockfish identify critical points in the game and decide when to spend some extra time on a move; at a fixed time per move, the strength will suffer significantly). The version of Stockfish used is one year old, was playing with far more search threads than has ever received any significant amount of testing, and had way too small hash tables for the number of threads.

To DeepMind's credit, they fixed all these weaknesses in the final published paper.

So yes, I would do it. Other parts of their review might be less reliable (I would certainly examine them carefully), but the person will be a great expert on whether the company's product is being used correctly.

Caveat: the person/company might be more incentivized to take action based on the unpublished results. For example, they might seek to improve their product using the methodology of the paper before it is published. I don't think there is a good solution to this because it can happen regardless of who the reviewer is; I would therefore do it anyway.