What should you do if you cannot reproduce published results?

Question

I am trying to reproduce published results in a paper. Those results come from numerical simulations. The original authors and I do not use the same software, and theirs is proprietary (I don't have access to it). I have tried to reproduce their results, and it works qualitatively but not quantitatively: the differences between their results and mine on typical quantities of interest are between 2 and 5 times the expected accuracy of the method.

So far, I have communicated with the original authors, trying to clear out all possible sources of error I could think of (checked that I got the tricky parts of the algorithms right, checked that the “usual” parameters that were missing from their paper had the “usual” values, everything I could think of). They are forthcoming enough, and reply to my questions quickly, but it's clear they don't want to invest time in doing any serious follow-up on their side. And without access to their software, it appears I'm stuck.

Now, my question is on how to proceed. The “ideal case” for unreproducible results is to make a detailed analysis of how and why they cannot be reproduced, and possibly find out a source of failure (or at least plausible issues). This advances the field, and is probably publishable (not in a very high-profile journal). Here, this is not possible.

I have, however, nice results that I have obtained (extending their work far beyond what was already published), and if I didn't have these differences with their paper, it would make a very attractive paper. What can I do with those? Is it possible to publish them, merely noting the different with their paper without more comment? Or are my results simply unpublishable? I welcome any comment, especially from people who have found themselves in such an uncomfortable situation!

Answer 1

Just publish. Publish your attempts to replicate the findings, documenting the discrepancies, together with the nice results you've obtained by extending their work. Consider sending a draft to the original authors for their comments.

Answer 2

Publishing results that contradict previous publications can be awkward, but if you can show that your method is correct beyond reasonable doubt, then it shouldn't be a problem. No code is guaranteed to be completely free of errors and no result is guaranteed to be correct just because it is published.

You don't say much about the nature of your computations/methods, but do you have any test cases for which analytical solutions are known or can be derived? If you can show that your code reproduces these results, then you can make the case that your code's results for the specific problem in question should be reliable.

Ideally, if you have such a test-case, you could ask the other authors to run it with their own code, and see if they also produce correct results. They may not want to, but that's their problem, not yours.

In summary, if you go to reasonable lengths in your paper to demonstrate the accuracy of your code/method against known analytical solutions, you shouldn't be too worried about not matching other people's results. At least that would be my opinion as a referee.

Answer 3

(Disclaimer: I have no personal experience with such a situation, so I'm just going off plain common sense. That said...)

It sounds like you've already taken every reasonable step to discover the source of the discrepancy, and you're now left with just an "unexplained deviation" between your results and theirs. You also say that the discrepancy doesn't actually affect the qualitative conclusions drawn from the results in any way.

At this point, if I were you, I'd just go ahead and publish your extended results, and just briefly note the discrepancy when you compare your results with prior work. As long as you're reasonably certain that your results are correct (up to expected limits of numerical accuracy), you can't really be expected to be able to explain any inaccuracies in other people's results. Of course, you definitely should make sure that others can easily reproduce your results and verify the correctness of the methods you used to obtain them, e.g. by making your software freely available.

If you really think that merely documenting the discrepancy between the two sets of results would be publishable on its own, doing that and then citing that publication in your main paper could also be an option. Generally, though, I'd expect that to be practical only if the precise quantitative values in dispute are actually of importance to others working in your field.

Answer 4

All the publications in the scientific journals should be reproducible and accurate. It is very important task to examine others' results. Original authors get lots of credit if an independent researcher verifies their theory or model.

Almost all journals have a section named Comment, *Letters* or Letter to Editor. Below are some links to these columns:

• Comment in IEEE Trans. Antennas. Prop. A Comment on “Joint elevation and Azimuth direction finding using L-shaped array”

• Letters in PNAS Genome composition, caste, and molecular evolution in eusocial insects

• Correspondence in Nature Reply to 'Measurement of mobility in dual-gated MoS2 transistors'

  You can send your findings to the same journal and explain all of your finding. Then the editors send your comments to the authors, and they need to provide appropriate response to the journal.

Of course, this is tricky, and you need to be careful. If you think everything is precise in your code, comment on the paper is an option.

Answer 5

I would be very wary of publishing and as a reviewer I would be wary about recommending publication. The unexplainable difference in results hints at a mistake. That mistake is either yours or theirs. I would like to know for sure that it is their mistake before publishing. Even though you cannot compare the two methods directly, you could still publish your method independently showing that it gives the "correct" answer in a battery of test cases and then noting that it gives a different answer in the non-testable case.

You could then refer to this paper when you publish the real work. The advantage is that it removes the need to dilute the message of the real paper with the details of the method. A second advantage is it may result in the original authors running the test batter with their method. This is especially true if you call them out in an earlier draft and send them a copy prior to submission. You could also request them as reviewers.

A different strategy might be a lab visit (physical or virtual) to use their software on your test battery.

Answer 6

Can you/are you willing to throw a bone?

It might be worthwhile to discuss having them as an author on the paper. Perhaps you can strike a deal to get what you want (get access to their software, results, etc.) in return for including them in your publications and having some level of collaboration. You might be surprised just being honest about this and talking about it openly might work! My best two papers to date come from doing exactly that and then developing working relation with the people that were not that forthcoming. After that we have published two additional papers together. Who knows you might actually end up collaborating and doing bigger and better thing together if it works. I would give it a try before deciding to just publish the results the way you described.

Answer 7

I'd like to add one point that has not been mentioned yet. It may or may not be applicable to your situation, but it might be in the general case. You mention that the output from the numerical simulations don't agree. Therefore, I suggest:

If two models can't be made to agree, it's time to do measurements.

Actually, this is a good idea even if they do agree, but if you can do measurements, you will be able to confirm that at least one of the models is incorrect at least for the specific situation of the measurement.

Of course this is not always possible.

Answer 8

I agree with others that have suggested that publishing the new results are OK. Mention that there is a difference with the old method but that it is not qualitatively different.

Many journals have a policy of asking for a comment/rejoinder from the author of any study whose work directly contradicted. If the editors think your comment within the paper qualifies, you might finally get the answer you are looking for.

But I would also urge you to think hard about how much you want to put on the line for this discrepency and how much time you want to spend on it. It sounds like the results are qualitatively the same. If you end up being 100% correct on everything, the contribution from a paper that only talks about the difference will be a slightly better estimate. In some situations, that can be worth a lot. In lots of others, it doesn't count for much. You'll know how important this is for your field.

I once found a small methodological problem in a paper in my second year of grad school. I asked a similar question to yours to a professor who asked me if the methodological error was likely to change the result or invalidate the core findings. When I said it was very unlikely, he told me that it was probably not the best use of my time to work a lot on a rejoinder.

It's tough. I think you should say something. A note in the paper is probably enough. For this sort of thing, I think a research note on Arxiv that you can cite might be an alternative.