63

I am trying to reproduce published results in a paper. Those results come from numerical simulations. The original authors and I do not use the same software, and theirs is proprietary (I don't have access to it). I have tried to reproduce their results, and it works qualitatively but not quantitatively: the differences between their results and mine on typical quantities of interest are between 2 and 5 times the expected accuracy of the method.

So far, I have communicated with the original authors, trying to clear out all possible sources of error I could think of (checked that I got the tricky parts of the algorithms right, checked that the “usual” parameters that were missing from their paper had the “usual” values, everything I could think of). They are forthcoming enough, and reply to my questions quickly, but it's clear they don't want to invest time in doing any serious follow-up on their side. And without access to their software, it appears I'm stuck.

Now, my question is on how to proceed. The “ideal case” for unreproducible results is to make a detailed analysis of how and why they cannot be reproduced, and possibly find out a source of failure (or at least plausible issues). This advances the field, and is probably publishable (not in a very high-profile journal). Here, this is not possible.

I have, however, nice results that I have obtained (extending their work far beyond what was already published), and if I didn't have these differences with their paper, it would make a very attractive paper. What can I do with those? Is it possible to publish them, merely noting the different with their paper without more comment? Or are my results simply unpublishable? I welcome any comment, especially from people who have found themselves in such an uncomfortable situation!

9
  • 1
    You mention different software. How much difference between the hardware platforms?
    – Nobody
    Feb 10, 2013 at 9:18
  • 1
    That was my 2-cents worth.
    – Nobody
    Feb 10, 2013 at 9:28
  • 1
    Are you going to make your code freely available?
    – Nicholas
    Feb 11, 2013 at 9:30
  • 1
    Can you compare model results to measurements?
    – gerrit
    Feb 11, 2013 at 12:38
  • 6
    @F'x Great - so your code is freely available and you can publish (or at least make available) your input parameters. Interested workers in the field can then replicate your results easily and hopefully confirm them if desired. The same can't be said necessarily of the authors of the original paper owing to the proprietary nature of their software. I've added my vote to EnergyNumbers' answer. Publish.
    – Nicholas
    Feb 11, 2013 at 20:00

8 Answers 8

54

Just publish. Publish your attempts to replicate the findings, documenting the discrepancies, together with the nice results you've obtained by extending their work. Consider sending a draft to the original authors for their comments.

2
  • 9
    If I was the editor of the journal or conference this was submitted to, I would almost certainly invite one of the authors of the original work for a formal blind review. Feb 11, 2013 at 4:25
  • 3
    @ChrisGregg Wouldn't there be a conflict of interest for the reviewer?
    – Our
    Jun 22, 2020 at 10:52
19

Publishing results that contradict previous publications can be awkward, but if you can show that your method is correct beyond reasonable doubt, then it shouldn't be a problem. No code is guaranteed to be completely free of errors and no result is guaranteed to be correct just because it is published.

You don't say much about the nature of your computations/methods, but do you have any test cases for which analytical solutions are known or can be derived? If you can show that your code reproduces these results, then you can make the case that your code's results for the specific problem in question should be reliable.

Ideally, if you have such a test-case, you could ask the other authors to run it with their own code, and see if they also produce correct results. They may not want to, but that's their problem, not yours.

In summary, if you go to reasonable lengths in your paper to demonstrate the accuracy of your code/method against known analytical solutions, you shouldn't be too worried about not matching other people's results. At least that would be my opinion as a referee.

5
  • 1
    Contrast “if you can prove that your method is 100% correct, then it shouldn't be a problem” and “no code is guaranteed to be completely free of errors”. There is no way I can prove that my method has no bug.
    – F'x
    Feb 10, 2013 at 14:49
  • @F'x: Good point, I amended my answer accordingly :) What I meant is to show that it is correct beyond reasonable doubt.
    – Pedro
    Feb 10, 2013 at 14:54
  • 5
    @F'x This is a bit pedantic, but it code can be proven to be correct, although it's rare and usually happens on a small scale. Feb 10, 2013 at 22:46
  • @PeterOlson, your comment begs for the famous quote by Donald Knuth: Beware of bugs in the above code; I have only proved it correct, not tried it. ;)
    – Leo
    Feb 12, 2013 at 8:19
  • @Pedro Excellent idea on the benchmarking, which is really one of the best ways to validating the accuracy of a method. It's really amazing how often we researchers avoid this critical step, thinking we have everything in order when we don't.
    – che_kid
    Mar 8, 2013 at 3:30
12

(Disclaimer: I have no personal experience with such a situation, so I'm just going off plain common sense. That said...)

It sounds like you've already taken every reasonable step to discover the source of the discrepancy, and you're now left with just an "unexplained deviation" between your results and theirs. You also say that the discrepancy doesn't actually affect the qualitative conclusions drawn from the results in any way.

At this point, if I were you, I'd just go ahead and publish your extended results, and just briefly note the discrepancy when you compare your results with prior work. As long as you're reasonably certain that your results are correct (up to expected limits of numerical accuracy), you can't really be expected to be able to explain any inaccuracies in other people's results. Of course, you definitely should make sure that others can easily reproduce your results and verify the correctness of the methods you used to obtain them, e.g. by making your software freely available.

If you really think that merely documenting the discrepancy between the two sets of results would be publishable on its own, doing that and then citing that publication in your main paper could also be an option. Generally, though, I'd expect that to be practical only if the precise quantitative values in dispute are actually of importance to others working in your field.

8

I would be very wary of publishing and as a reviewer I would be wary about recommending publication. The unexplainable difference in results hints at a mistake. That mistake is either yours or theirs. I would like to know for sure that it is their mistake before publishing. Even though you cannot compare the two methods directly, you could still publish your method independently showing that it gives the "correct" answer in a battery of test cases and then noting that it gives a different answer in the non-testable case.

You could then refer to this paper when you publish the real work. The advantage is that it removes the need to dilute the message of the real paper with the details of the method. A second advantage is it may result in the original authors running the test batter with their method. This is especially true if you call them out in an earlier draft and send them a copy prior to submission. You could also request them as reviewers.

A different strategy might be a lab visit (physical or virtual) to use their software on your test battery.

5
  • The software is different, but the method is supposed to be the same.
    – F'x
    Feb 10, 2013 at 14:50
  • "The unexplainable difference in results hints at a mistake." Yes, it does. But it does not provide any particular handle on which party made it. F'x hasn't provided us with any particular reason to that that the earlier publication can be taken as gospel. A important question here is "Have other reproduced it?". Feb 10, 2013 at 23:11
  • 9
    Any concerns that @F'x is the one making the mistake could be softened by F'x making his code publicly available.
    – JeffE
    Feb 11, 2013 at 5:41
  • I like the idea of giving them the chance to run your tests with their tool. If they get bad results there, you can comfortably call the former results errorneous. (Provided your tests are good, of course.) If they get good results, you are back at square one.
    – Raphael
    Feb 11, 2013 at 12:22
  • @F'x Clearly the software and the method are not the same . This could be different algorithms or different settings/parameters between the two codes. I agree with Daniel E. Shub that I would (as an editor or reviewer) be hesitant to accept your results without some attempt to explain the discrepancies. For better or worse, their work is published and yours isn't so the responsibility lies on you to try and explain the differences, or at least validate your method.
    – che_kid
    Mar 8, 2013 at 3:38
8

All the publications in the scientific journals should be reproducible and accurate. It is very important task to examine others' results. Original authors get lots of credit if an independent researcher verifies their theory or model.

Almost all journals have a section named Comment, *Letters* or Letter to Editor. Below are some links to these columns:

Of course, this is tricky, and you need to be careful. If you think everything is precise in your code, comment on the paper is an option.

7

Can you/are you willing to throw a bone?

It might be worthwhile to discuss having them as an author on the paper. Perhaps you can strike a deal to get what you want (get access to their software, results, etc.) in return for including them in your publications and having some level of collaboration. You might be surprised just being honest about this and talking about it openly might work! My best two papers to date come from doing exactly that and then developing working relation with the people that were not that forthcoming. After that we have published two additional papers together. Who knows you might actually end up collaborating and doing bigger and better thing together if it works. I would give it a try before deciding to just publish the results the way you described.

2
  • 5
    This can be a bit tricky... The joint paper may just as well look like the original author extending his/her own work, with you as just a subordinate co-author. The question you have to ask yourself is: Will this be seen as my own work/contribution? This is the question that hiring/grant committees will be asking too. If it is not clear that it's your own work, e.g. the other author is more senior and still active in the field, and/or he/she will present the joint work at conferences and talks too, then stay away from it.
    – Pedro
    Feb 10, 2013 at 12:28
  • @Pedro I suggested because "It's clear they don't want to invest time in doing any serious follow-up on their side". I agree that this could happen if the relation is not managed and negotiated well. But the up side is also significant as i experienced. In my case it was with the world lead author in the field and yes he took some credit for it but always mentioned me. I have got four papers out of working with him so far (and we still going) all in the fields top journals that i could publish and i think this had to do with the fact that i was publishing with him...
    – blackace
    Feb 10, 2013 at 22:28
7

I'd like to add one point that has not been mentioned yet. It may or may not be applicable to your situation, but it might be in the general case. You mention that the output from the numerical simulations don't agree. Therefore, I suggest:

If two models can't be made to agree, it's time to do measurements.

Actually, this is a good idea even if they do agree, but if you can do measurements, you will be able to confirm that at least one of the models is incorrect at least for the specific situation of the measurement.

Of course this is not always possible.

5

I agree with others that have suggested that publishing the new results are OK. Mention that there is a difference with the old method but that it is not qualitatively different.

Many journals have a policy of asking for a comment/rejoinder from the author of any study whose work directly contradicted. If the editors think your comment within the paper qualifies, you might finally get the answer you are looking for.

But I would also urge you to think hard about how much you want to put on the line for this discrepency and how much time you want to spend on it. It sounds like the results are qualitatively the same. If you end up being 100% correct on everything, the contribution from a paper that only talks about the difference will be a slightly better estimate. In some situations, that can be worth a lot. In lots of others, it doesn't count for much. You'll know how important this is for your field.

I once found a small methodological problem in a paper in my second year of grad school. I asked a similar question to yours to a professor who asked me if the methodological error was likely to change the result or invalidate the core findings. When I said it was very unlikely, he told me that it was probably not the best use of my time to work a lot on a rejoinder.

It's tough. I think you should say something. A note in the paper is probably enough. For this sort of thing, I think a research note on Arxiv that you can cite might be an alternative.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .