I obtained the code for a published article from its author and found major mistakes in it. What should I do?

Question

I contacted an author to obtain his data and code for a published paper because I could not figure out how he was obtaining some results, given that I was using the same dataset. Once I obtained the code, I started digging in and I found out that there are many mistakes leading to the published results. Or at least to the credibility of said results, since a major assumption is not satisfied as claimed in the paper.

What is the best practice in this case?

If the data and code were publicly available, I would be more prone to send a comment to the journal but since I obtained them directly from the author, I feel I would kind of violate his trust (even though the data policy of the journal is public availability through the authors). At the same time, this is one of the few results answering a specific question and I think it is misleading to see it published and cited.

Answer 1

If the paper is minor and of little consequence, there is probably nothing to do. And if the errors are relatively minor and easily corrected, then it is likely that few will care as the ideas themselves might shine through. But that is a bit subtle.

Otherwise, I'd suggest that the best course of action is to let the original author know of the problems, say that you want to provide a follow up - corrected - version, and invite collaboration on it.

The paper will be easiest to write as a collaboration than as a follow on since you otherwise need to be very careful about the possibility of plagiarism. It can be done, but it makes the result a bit more awkward.

But that assumes that the results you obtain will be significant enough to be published.

Answer 2

Your main duty is to the broader scientific community, which should be presented with clear and correct results.

You should surely contact the authors of the wrong article. But you should cooperate with them only insofar as they really want to correct their mistakes.

Writing to the journal is possible and not at all a breach of trust, however beware that in practice journals can be reluctant to admit mistakes, may take a long time to do so, and may end up minimizing the issue. ("Minor technical glitches, conclusions unaffected.")

The best course of action depends on the particular field. If you are in a "healthy" community, cooperating with the authors may be best. If you are in a less healthy situation, you may have nothing good to expect from the authors and journal, and it would be better to raise your concerns directly to the public in a preprint or a PubPeer comment. You probably need advice from someone in your field.

Answer 3

You should contact the author of the paper first, and ask some questions. These questions should be non-judgmental. The problems you found with the paper may not be known to the author, nor was it their intention to publish with such errors present. You have not given details regarding the paper, or what can be determined in the course of events regarding that author's research. But your totally objective and unbiased questions regarding the results may bring to light problems that this author is now seeing for the first time. They may wish to correct these mistakes. This may open a collaborative effort that ultimately becomes quite productive and rewarding.

What else can you do? Put yourself in their shoes, and presume at the start they do not know of these problems. State you found this author's paper interesting regarding a problem in the nexus of your own ongoing research. You do not have to reveal more than necessary (in asking some questions) other than to say you are doing some background research (you may, or may not, be at liberty to discuss). See what they have to say, and note how they respond. You can set boundaries regarding how much you wish to discuss, and see how things develop from there. But be honest. Trust will come but not all at once. Keep in mind, this author may have adverse feelings about your asking questions regarding noted errors in this paper. Do they know you and can they trust you? They may get defensive, and you should be prepared for this. You may have to let go of discussing this issue, altogether.

Contacting the journal may not be a good idea. What are they supposed to do? A better way to approach this problem is in the publication of your own research. If the effort is collaborative with this author, then all is well. But, if you have to go it alone, do not directly mention that errors were found in another's work, or be critical. Rather, a more or less casual statement that is to the point, for example, would be to state that among papers X and XY discussing this problem, the current work provides an updated view of the analysis and results. No judgements, no discussion of errors or criticisms, just some objective and forward-looking comments and statements. Keep in mind, the author of the work we are discussing may be waiting for your work to be published. There is no reason that this author should feel badly about comments you may have regarding their work. There is no reason to leave adverse consequences for anyone.

Answer 4

First: you are not a policeman, nor a judge looking to ascertain if anyone is guilty/innocent of whatever crime.

Write the authors, stating that with the provided code you are not capable of replicate the results.

Show a collaborative attitude. If they have a similar attitude, you may end up writing an errata on the same paper with the original authors.

If they are not cooperative, it is time to try to get published a comment on that paper. This do not imply stating anything about their intentions, just that the results are not reproductible.

Final note: writing a commentary paper is even harder than the paper itself. Be ready to endure a long path ... or try to publish independently on the same topic so the state-of-the-art goes beyond what is stated in the bugged paper you found.