What to do when publishing a paper about mistakes in previous literature?

Question

Two years ago I read a paper in which the "Methods" section did not report enough information to reproduce the analysis presented by the authors.

I contacted the first author in order to have more information and he provided me the code he used to analyse the dataset presented in that paper.

I found out that in the code there was a conceptual mistake in the way some algorithms were used. However, I did not have a solution for the mistake. So, I just ignored the fact and started working on a different problem.

One year later, while working on the new problem, I found a solution that could permit to avoid the conceptual mistake that I found in the above-said paper. As in the meanwhile I stumbled across to another paper doing the same methodological mistake, I thought to write a letter to the journal in which the first paper was published to underline the problem in those two papers and propose a solution. The paper analyses a simple dataset in order to show the differences between the results obtained with the old method and with my new method. I also (of course) point to the fact that the data presented in the two papers should be re-analyzed, as a data analysis containing that mistake can produce misleading results.

The peer-review process went smooth, with constructive comments of the reviewers. At the end of this process the Editor informed me that they also requested a signed review by the first author of the paper published in his journal. So, the same person who originally provided me his code.

The recommendation that he gives on my paper is of major revision. His comments are (in principle) constructive but he requires me to cancel from the paper the reference to his paper because: i) the re-analysis of his dataset with my method (re-analysis that he did not share) does not show significant differences; and ii) the criticism of his results is not justified as the Methods section in his paper does not contain sufficient details to infer the error. Do you think that these request are justified?

During a private conversation with me he also made clear that it is his interest to make sure that his original paper "would not be destroyed" and that he does not consider to publish a re-analysis of his data. This is particularly nonsensical because the results affected by the error are just a rather small part of the results presented in his paper and could only be found out by inspecting the code.

Given this context, I think that his requests are quite unfair. It is clear that he is just trying to preserve his publication record. It also makes clear that all his positive comments in the review are just an attempt of bargaining a positive recommendation for the publication of my paper in exchange of erasing the reference to the error in his paper.

Therefore, my conundrum is if I should just give in to his requests for an easy publication (nice thing during a PhD) or do not bend, argument against his comments and get a rejection recommendation. What would you do?

Answer 1

To begin with, I find it highly problematic that this journal has a policy of inviting authors to review any paper that criticizes their work. That seems to me to be highly inhibitory of criticism and likely to produce situations exactly like the one that you are facing.

I find it doubly problematic that this author has chosen to communicate with you outside of the review process in order to persuade you to change your writing. His communications can easily be interpreted as an unethical quid pro quo request: you drop your criticism of his paper, and he'll drop his criticism of yours.

What is not clear from what you have written, however, is just what the journal intends to do with the review that they have received. Surely the journal must understand the potential conflict of interest here. So is the review mainly to allow an author to point out errors in the critique that those less familiar might overlook, or are they going to treat it just like one of the other reviews?

I would thus recommend taking the following approach:

1. Write to the editor and let them know that the author has approached you separately with what you are concerned may be an unethical proposal --- the editor needs to know about this and should be able to make their approach clear to you. Ask the editor how they would like to proceed.

2. In parallel, use all of the constructive parts of the criticism to improve your manuscript.

3. Consider slightly softening the language in your criticism in the manuscript. I do not know how you have written it, but it's often possible to use slightly less definitive language to make the same point, e.g., changing "Papers [1] and [2] should be re-analyzed to see if their results are affected." into "As it is possible that results might be affected, we would recommend considering re-analysis of papers [1] and [2]." Same substance, but less confrontational and allowing you to write that you have adjusted the paper in response to the review.

Hopefully, the editor can provide you some clarity as to how to navigate the difficult situation that their journal has created for you, and judicious adjustment can allow you to pass peer review without compromising your ethics. If not, there are other good journals out there that will not have this unusual conflict-generating policy.

Answer 2

In case of a corrigendum there is a standard practice to ask the author(s) of the original paper (OA, original authors). However, ignoring the "opposed" request is highly problematic assuming the journal offers such a list. Take a different journal, write the new editor that you will put OA into the opposed list, and ask (do not request) the editor whether such a list would be observed. State that if it won't, you'll withdraw the paper, and that you'll wait for a clear yes/no before submitting. Yes, and all your e-mails should be utmost polite, but still clear.

Your re-submission should be improved in any case. Use the comments of the OA. Cite OA, scold his technical results, but acknowledge himself a lof for his source code and reviews.

If everything fails, consider a journal of negative results. There are some of them around in the meantime.