5

I contacted an author to obtain his data and code for a published paper because I could not figure out how he was obtaining some results, given that I was using the same dataset. Once I obtained the code, I started digging in and I found out that there are many mistakes leading to the published results. Or at least to the credibility of said results, since a major assumption is not satisfied as claimed in the paper.

What is the best practice in this case?

If the data and code were publicly available, I would be more prone to send a comment to the journal but since I obtained them directly from the author, I feel I would kind of violate his trust (even though the data policy of the journal is public availability through the authors). At the same time, this is one of the few results answering a specific question and I think it is misleading to see it published and cited.

2
  • "...since a major assumption is not satisfied..." I have too little to add to the existing answers to write my own. I just would like to mention that an assumption which is not satisfied does not automatically lead to wrong conclusions. This very much depends on the actual case. Nov 29, 2021 at 17:31
  • That's fair. I didn't want to get too deep into technicalities but if you make this assumptions, results are valid. If the assumption is not satisfied, the results are not reliable at all. It's a statistical assumption.
    – PhDing
    Dec 1, 2021 at 17:07

4 Answers 4

2

If the paper is minor and of little consequence, there is probably nothing to do. And if the errors are relatively minor and easily corrected, then it is likely that few will care as the ideas themselves might shine through. But that is a bit subtle.

Otherwise, I'd suggest that the best course of action is to let the original author know of the problems, say that you want to provide a follow up - corrected - version, and invite collaboration on it.

The paper will be easiest to write as a collaboration than as a follow on since you otherwise need to be very careful about the possibility of plagiarism. It can be done, but it makes the result a bit more awkward.

But that assumes that the results you obtain will be significant enough to be published.

1
  • Thanks for the suggestions! The journal is not very relevant in the field but the paper is one of the two published on the topic in a specific geographical setting so it comes up immediately when googling the right keywords.
    – PhDing
    Nov 27, 2021 at 22:02
1

Your main duty is to the broader scientific community, which should be presented with clear and correct results.

You should surely contact the authors of the wrong article. But you should cooperate with them only insofar as they really want to correct their mistakes.

Writing to the journal is possible and not at all a breach of trust, however beware that in practice journals can be reluctant to admit mistakes, may take a long time to do so, and may end up minimizing the issue. ("Minor technical glitches, conclusions unaffected.")

The best course of action depends on the particular field. If you are in a "healthy" community, cooperating with the authors may be best. If you are in a less healthy situation, you may have nothing good to expect from the authors and journal, and it would be better to raise your concerns directly to the public in a preprint or a PubPeer comment. You probably need advice from someone in your field.

1
  • I was planning to talk to my advisor before doing anything but I thought getting some broader advice could have been helpful. Thanks!!
    – PhDing
    Nov 29, 2021 at 14:43
0

You should contact the author of the paper first, and ask some questions. These questions should be non-judgmental. The problems you found with the paper may not be known to the author, nor was it their intention to publish with such errors present. You have not given details regarding the paper, or what can be determined in the course of events regarding that author's research. But your totally objective and unbiased questions regarding the results may bring to light problems that this author is now seeing for the first time. They may wish to correct these mistakes. This may open a collaborative effort that ultimately becomes quite productive and rewarding.

What else can you do? Put yourself in their shoes, and presume at the start they do not know of these problems. State you found this author's paper interesting regarding a problem in the nexus of your own ongoing research. You do not have to reveal more than necessary (in asking some questions) other than to say you are doing some background research (you may, or may not, be at liberty to discuss). See what they have to say, and note how they respond. You can set boundaries regarding how much you wish to discuss, and see how things develop from there. But be honest. Trust will come but not all at once. Keep in mind, this author may have adverse feelings about your asking questions regarding noted errors in this paper. Do they know you and can they trust you? They may get defensive, and you should be prepared for this. You may have to let go of discussing this issue, altogether.

Contacting the journal may not be a good idea. What are they supposed to do? A better way to approach this problem is in the publication of your own research. If the effort is collaborative with this author, then all is well. But, if you have to go it alone, do not directly mention that errors were found in another's work, or be critical. Rather, a more or less casual statement that is to the point, for example, would be to state that among papers X and XY discussing this problem, the current work provides an updated view of the analysis and results. No judgements, no discussion of errors or criticisms, just some objective and forward-looking comments and statements. Keep in mind, the author of the work we are discussing may be waiting for your work to be published. There is no reason that this author should feel badly about comments you may have regarding their work. There is no reason to leave adverse consequences for anyone.

1
  • Sure, I did not plan of accusing anyone of cheating. It is possible that the mistake just happened or there was lack of understanding of the specific method used. Thanks for the feedback!
    – PhDing
    Nov 27, 2021 at 23:09
0

First: you are not a policeman, nor a judge looking to ascertain if anyone is guilty/innocent of whatever crime.

Write the authors, stating that with the provided code you are not capable of replicate the results.

Show a collaborative attitude. If they have a similar attitude, you may end up writing an errata on the same paper with the original authors.

If they are not cooperative, it is time to try to get published a comment on that paper. This do not imply stating anything about their intentions, just that the results are not reproductible.

Final note: writing a commentary paper is even harder than the paper itself. Be ready to endure a long path ... or try to publish independently on the same topic so the state-of-the-art goes beyond what is stated in the bugged paper you found.

4
  • 1
    As far as I understand, the OP is able to reproduce the published results with the provided code. Nov 29, 2021 at 17:24
  • @Snijderfrey code is not just the actual code. Otherwise you could just port a code to another format and it will not be plagiarism. The code is also the assumptions behind the code itself: as every programmer knows, a machine is fast but stupid, a certain code is just the execution of many instructions and the instructions are given by the coder. Assumptions are not respected --> the paper has "bugs".
    – user149718
    Nov 29, 2021 at 22:06
  • I don't want to disclose too many information because the topic is sensitive but it's code to produce statistical analysis so I can tell if the code is correct or not based on what I read. I am no policeman and maybe the authors simply did not have the knowledge, I am not accusing anyone here. Just wondering if I should do something.
    – PhDing
    Dec 1, 2021 at 17:05
  • 1
    @PhDing There can be a number of errors, as far as I understood the code produces the results, but the code does not respect the assumptions presented in the paper (roughly like "parameter a is in the range 0 to 0.5" and then in the code a = 0.75). If the authors are not cooperative, you may repsect the assumptions, run the code, check the results, if they do not match the paper results, go down the rabbit-hole of contacting authors that cited that paper (maybe you met some of them at conference/workshops?), bringing to their attention the errors and finding some support.
    – user149718
    Dec 2, 2021 at 10:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .