Is it reasonable to cite published computational results rather than reproduce them yourself?

Question

In computer science, when we write an academic paper, we often have to include comparative results that our method has some kind of quantitative advantage against some other state of the art results.

While it is sometimes easy to reproduce a cited paper's results (e.g., classification of small datasets), other times it is very time consuming and you could spent the better part of your research time just writing code testing the other methods.

However, it sometimes happens that when we reproduce the method, we do not get the presented results. So it can be tricky to do that as well.

• Is it reasonable just to cite another paper's results at face value?
• What suggestions would you give to this conundrum?

Answer 1

• I treat computational results as in silico experiments. They have been published in a peer-reviewed journal, so I trust them by default, unless I have a reason not to. I cite them without a need to recompute them.
• Sometimes, I have a reason to doubt them: they don't match my intuition, or they don't match my own results in a related case; they seem incoherent; they don't match experimental data; etc. Then, I redo them, possibly in more than one way (different software, try checking the effect of some of the assumptions, etc.). If I learn something from it, I consider publishing this study.

Answer 2

My most-cited work arose because we couldn't reproduce an earlier published paper in the field. We did a much more extensive study, and demonstrated conclusively why the previous results were unreliable.

So, while you don't need to reproduce every single result that has previously been published, it can serve a useful purpose to try to reproduce at least some of those results—because then you can be sure that your model is working the way you would expect it to (provided the previous data can be trusted; as my case showed, this isn't always the case!).

Answer 3

Is it reasonable just to cite another paper's results at face value?

In my opinion, there are only few limited cases where this would be acceptable. For example, when comparing industrial-scale systems, or comparison of qualitative features of the works. Also in the case there exist an established set of benchmarks and your algorithm can solve some of those the other can't, you do not really need to reimplement. Possibly also when testing the algorithm cannot be done in a reproducible manner (industrial-scale field tests). When it comes to experimental efficiency, however, I think you usually do not have much choice and should reproduce the others results - if possible at all.

What suggestions would you give to this conundrum?

Depends on what you need the cited work for. In the case you developed an algorithm which is supposed to be more efficient than the cited one, to prove your point you need a controlled experiment when both methods are run under same conditions (e.g., implemented in the same programming language, and run on the same system etc.) and with the same set of benchmarks. In such a case, if you want to demonstrate your point properly, your best is to reimplement (or ask the authors for their implementation and adapt it to your conditions) and run head-to-head. Otherwise your performance curves are going to be incomparable. For good examples of how to do this, see for example comparisons/evaluations of experimental performance of planning algorithms, or SAT solvers.