They shared the code, but then, I found key errors in their work. How do I criticize this in my research paper?

Question

Associate Prof. X is committed to reproducible research and makes his code available. He authors a paper with Dr. Y and others.

Dr. Y is the first author of the paper and the code is shared on Ass. Prof. X's website. It is clearly stated in the paper that the code produces the graphs in the paper.

I downloaded the code, ran the code and found it works well i.e produces the same graphs in the paper. However, the code was not descriptive and easy to understand (Apparently, the variables which makes it easy to read has been changed to something difficult to understand except with much effort and time). Anyway, I studied the code and was able to clearly understand the meanings of various parts of the code. I gave the variables their proper definitions while generating the same graphs

Now, I find the graphs for some figures are not provided, then mailed Ass Prof. X, who after a while informed me that first author Dr. Y will send the code. The code was sent and I was able to understand this paper.

Currently, I am about making an extension to this work. However, I observe there are some errors in several equations in the paper which adversely affects the behavior of some graphs! Furthermore, on a deeper investigation into the paper, some inferences and propositions were made based on some of the mistakes within the paper! At a time I tried contacting Dr. Y about something I wanted to know better about the paper but I didn't get a response.

The challenge I have now is, I want to present my own paper and as well make corrections to these mistakes since it directly extends this work. How do I handle this? Do I say for example: "We corrected equation (y) in [ref] and thus, the graphs behave like this?". Also, I am thinking that since I got the code for some of the graphs (via email), I should thank the authors for sharing the code (for those graphs). However, I am not sure this is appropriate as the public gets to know implicitly that the code for some of the graphs are not provided in the paper.

Note that, I appreciate and consider it as an obligation to be straightforward in my research!

Answer 1

I would think several times before using the word 'corrected', especially if I don't fully understand the code, as it seems from your question ("...I studied the code and was able to decipher the meanings of various parts of the code."). Understanding various parts is different from understanding it entirely, the latter being a prerequisite to improving upon it or correcting it.

However, if you are sure that mistakes exist in the code, you have two options:

(a) Point out those mistakes and share them with X and Y before trying to publish it. Once you publish, the credibility of their code goes down - if some parts are wrong others could be too. Therefore it is only fair that you convey the errors to them first. You aren't obligated to, but I think it's the right thing to do.

(b) If you're significantly extending the code, write your own- it could be based on their code, and you can acknowledge that, but by writing your own code you take full responsibility for it, and are not propogating any errors inherent to the original code. In terms of quantum of work, I think this is more significant, and gives you more room to develop it for your particular purpose.

Answer 2

Firstly, yes, you should definitely thank them for providing their code, and making their research reproducible. This is extremely valuable, and it is precisely the ability to spot and correct errors in research that makes this a good practice. If it were me, I would be quite profuse in my thanks for this, and make sure that any criticisms of errors are contextualised by the excellent work done in preserving their data and code.

The other thing you should be careful of is to make absolutely sure you're right about the errors. I would suggest that prior to submission of your article you first contact the authors of the other work and show them the errors you have found in their work. This will give you feedback to confirm/deny the error. (You might even find it is something simple like they sent you the wrong version of the code.) Most authors are quite reasonable in responding to things like this, and I'm sure they will want to check their work to see if there are errors. If you find them responsive to your concerns, you might even consider suggesting these authors as referees for your paper.