How to proceed when the baseline (state-of-the-art) published results claim much better performance than I can reproduce?

Question

I am graduate student, to finish my degree I need to build methods outperform what is already there. An issue that I came across with, is that two papers reported way (I mean more than 20%) more than what resulted from my reimplementation. This could be due to two reasons:

1. I missed something during the implementation. Which is what I have been telling myself. For months, I tried all possible combinations and possible paths. One of the methods is straightforward. Still, I could not reach their claimed performance.

   I contacted the corresponding authors, and no one replied. So I tried to contact the other authors.

   The first paper, the author replied and sent me the code. He/she told me to keep all details ”confidential”. Well, it turns out they they are not using the data they claim in their the paper, of course their results are different than my reimplementation. And my implementation was correct.

   The second paper author also replied and they didn’t send me the code because they say it is easy to implement, but confirmed that what I did is correct still I couldn’t understand why such difference.

   Both papers are published in <2 impact factor journals. Their web servers are not working.

2. They are not honest.

Now I am stuck, my method does outperform my reimplementation of their methods but not what they claim. The first paper I can’t say anything because “it is confidential” the second paper I can only confirm that I correctly implemented their method for the most part (based on my chat with the authors)

I know that I probably could not publish on this part of my work, because who is going to believe a young scientist who just started her way? But not sure how the committee are going to believe me. What can I say or do? Please help me

Answer 1

There is absolutely no reason that you can't publish a paper that says "We compared our method to methods X and Y. Since code the original code was not available for X and Y, we reimplemented the methods to the best of our ability. The code for these reimplementations is available in supplementary files A and B. Our new method out performed the reimplementations of X and Y by z%. However, it should be noted that it was not possible to reproduce the reported results for X and Y. "

People who want to know will have to look at your re-implementations and decide themselves if they think you have correctly re-implemented.

Seniority has nothing to do with it - be transparent, and the world will judge if they believe you or the people that won't release their code.

Answer 2

People can be dishonest. They can also make honest mistakes and publish bad science. Don't assume that it is you who has an inferior result. And don't assume that a doctoral committee won't believe you. If they are competent to judge you without the earlier results they should be competent to understand what you have done.

However, I have two suggestions. The first is to walk through what you have done with your advisor and/or another faculty member who is most competent to understand your work. You may, indeed, have the best results. If you can get support there, then the larger committee should be no problem. I don't think that you need to hide the communication you got from your committee members. It may be necessary to explain why you can't believe the reported results from the other paper. I don't think that "confidentially" really applies here.

But the other is a bit harder. See if you can figure out exactly where the other group failed to match their methods to their results. If you can do that, then you have much stronger evidence for your own work.

The evidence you mention here seems pretty strong to me (an outsider) that the other paper has a problem. There is no reason not to contradict it if it is incorrect, for whatever reason.

Answer 3

to finish my degree I need to build methods outperform what is already there

No, that is not true. You need to deliver a piece of proper scientific work and advance knowledge and that does not depend on what direction your findings point.

Of course, things are easier and more pleasant if your implementation is better. But the actual scientific part of your thesis is to study both the old and your approach scientifically and then conclude whether one is better (and possibly in which situations).

The difficulty in your situation is to proove that the discrepancy to literature is not due to your incompetence or lack of hard work (=> you deserve a bad mark) but actually due to "nature" not being as it was supposed to be by the previous paper.

What you can and should report is

• that you were not able to reproduce the findings in papers 1 + 2,
• in consequence have been in communication with the authors.
• Importandly, that your implementation has been confirmed as correct by private communication with the authors of paper 2 and by comparison with (confidential) code you received from the authors of paper 1 again by private communication for that purpose.

• If

  Well, it turns out they they are not using the data they claim in their the paper, of course their results are different than my reimplementation.

  means that you got the data set they actually used and got the same results with that, then you can also report that for a related data set, the same results were obtained.
  If not, it may be possible to kindly ask the authors of paper 1 + 2 whether they'd run a data set you send them and give you the results of their implementations so you can compare that to your results. You can then report (hopefully) that equal results were obtained on a different data set and thank the authors of those papers for running your data.

The last two points should make amply clear that the discrepancy is not due to a fault in your implementation - which is what counts for your thesis.

As a personal side note, I got top grade on my Diplom (≈ Master) thesis which (among other findings) found that the software implementation I was using did not work as it was supposed to. I was able to point out a plausible and probable reason for that bug (which may have been a leftover debugging "feature") - which is much harder for you as you don't have access to a running instance of their software that you can test (= study) to form and confirm or dismiss hypotheses about its behaviour.

---

As an addition to what @Buffy explained already about the possibility of honest mistakes in published papers:
As scientists we tend to work at the edge of what is known. Which also means that we're inherently running a high risk of not (yet) knowing/having realized important conditions and limitations of what we are doing.
We thus also run a comparatively high risk that tentative generalizations we consider may turn out to be not all that general after all. Or that we may be plain wrong and realize this only later (or not at all). I believe it is very hard for humans to be completely aware of the limitations of the conclusions we draw - possibly/probably because our brains are "hardwired" to overfit. (Which also puts us into a bad starting position for avoiding overfitting in e.g. machine learning models we build)

The take-home message from this is that we need to be careful also when reading published papers: we need to keep the possibility of the paper being wrong, containing honest mistakes or not being as directly applicable to our task at hand as we believe at the first glance in mind.

---

I missed something during the implementation.

I experienced something similar once when I was also implementing a reference method from literature (related but different field). It turned out that different defaults in the preprocessing of the data caused the difference - but only after I had the bright idea of trying out to omit a preprocessing step - although the model doesn't make much sense physically without that step, but the paper didn't mention any such step (neither do many papers in my field who do use that step because it is considered necessary because of physics).

---

2. They are not honest.

While that is of course possible, I've seen sufficient honest mistakes to use Hanlon's razor (which I first met as Murphy's razor): and not assume dishonesty or misconduct unless there are extremely strong indications for that.

---

Proving superiority may in any case be something that is impossible due to limitations in the old paper.

E.g. if they report validation results based on a small number of cases, the uncertainty on those results may be so large and thus it cannot be excluded that the method is better than it seemed that truly improved methods later on will not be able to demonstrate their superiority in a statistically sound manner.

Still, such a shortcoming of the old paper does not limit the scientific content or advance of your work.

Answer 4

You can write that you used your implementation of the competing method for your results, and that you were not able to reproduce the published results. Make your code available so people can check.

It seems that the authors of the other papers didn't publish their code, so nobody can say you should've used that.

Answer 5

In the first place, you should consult this with your supervisors. The code for papers is often rushed and unfinished, and what works on one machine may not work on another for a number of reasons. The most reasonable way is to let your supervisors know that you implemented both methods, communicated with original authors (mention only non-confidential things/say some things are confidential/ask authors for permission to discuss the implementation with your supervisor), and yet you did not reach the performance claimed. As a senior academic capacity, they are better equipped to decide what to do with regards to politics of department/field/research teams, are bound to get quicker and more elaborate responses from authors of the papers and handle potential fallout should anything go wrong in the process. I would not advise to pursue this matter on your own, and surely if you have doubts about something this important to your project, it would be reasonable to seek their advise and they will understand that.

Answer 6

In addition to the other answers, you should consider publishing your re-implementation. Then any reviewers can check if they think your results are plausible or if they spot a flaw in your re-implementation.

In the first case, it is right to say "We implemented paper X, but could not reproduce the claimed efficiency" and in the second case the flaw found by the reviewer may help you to improve your re-implementation, so you achieve a similar result.

Most reviewers will not debug your code, but you did your best efford to allow anyone to verify your claims of less efficiency and at least your paper is as honest as possible.

If the algorithm is interesting, publishing an open source version may get some users, that point out issues with your code (or contribute improvements) as well. But make sure not to be too close to the confidential code, as the original authors may claim copyright infringement.

You may use clean room reverse engineering with another person or at least do it yourself by just using the given code to write down the parts missing in the paper and then reimplement it from the documentation and not from the code.