Should I compare my proposed work with my implementation of the previous related papers or the reported result?

Question

In computer science, I need to compare the accuracy of my proposed method to the previous ones. There are two possible options:

1. Implement the methods described in related papers on my own machine
2. Use the claimed results of the authors

Sometimes, when I implement the other papers, the results are different from the claimed ones, because of a lack of details, parameters and hyperparameters. What is the correct decision? Should I compare my work with the implemented one? In addition, sometimes the reports of the previous works are on a different dataset and I have to implement the code. Is it the correct way?

Answer 1

I'd say, as a rule you just rely on the results provided by the authors of the papers you cite. You aren't really supposed to reimplement their algorithm just to confirm their results.

Let's dive a bit deeper, especially from a reviewer's point of view. When a reviewer sees that your method in a certain way superior to a previously published work, this is exactly what is expected, so everyone is happy. If you can't prove that your method is better because you use a different dataset, it's quite a problem (for both you and the reviewer), and in general case it would be very advisable to get the same data. Perhaps, this is the first step to take.

If you can't get the same dataset and try to reimplement the algorithm, of course it opens the whole can of worms, since the reviewer might rightfully presume that your implementation is somehow different (due to different parameters or possible bugs) and thus incomparable. I'd say that a common valid scenario for such a venture is when you doubt the original paper's conclusions and want to confirm or disprove them.

As a side note, I think one should decide on the evaluation strategy before designing and implementing own methods. If you want to advance the state of the art, it's reasonable to check the existing papers first and make sure that you can get the same dataset as their authors to show that your method is more accurate on the same data.

Answer 2

1) Comparing with results that other authors report is in all likelihood acceptable.

2) However, published results shouldn't by uncritically believed, so it is better (in the sense of being a better service to science) to replicate other authors' results. Also, who knows, in case reported good results do not replicate you have another argument against a competitor and in favour of your method.
If information to do that is not sufficient in the paper you're citing (which reviewers of that paper should have criticised in my opinion), the most reliable way of doing that is to ask the authors for their code (if it isn't available anywhere anyway, that is).

3) If the authors don't share their code it is a good thing in my view to try to replicate their results using your own code (although in case you can use published results it is not mandatory for your publication, see point 1). If you find differences, it's best to contact the original authors about this, but you are also well within your rights to say in your paper that these results deviate from the original ones despite making your best attempt to replicate them. In any case you should acknowledge that this is your implementation, and list any decisions made by you for the implementation that are not obvious from the original paper. Once more, this is more work than just using their reported results and in all likelihood not required for publication, however the reward is a certain chance to find something that you can use against a competitor (I of course mean this in a purely peaceful way, but implicitly assuming that you want to write a paper and readers and reviewers may want to know what makes your method worthwhile compared with XXX).

I add (belatedly) that apart from the value for the specific publication you may have in mind, I learned a few valuable things trying to replicate other people's work.

4) The other respondents are right about only comparing what's comparable, see their answers.

Answer 3

This would depend on how you are measuring things. If the results are simple timings then machine speed/architecture can matter (a lot). But if you are, for example, counting comparisons, then the result is independent of machine speed.

Be sure that you are comparing apples to apples and not apples to peanuts.

Answer 4

Computational complexity is a useful tool for measuring performance and comparing results. For instance, sorting algorithms are compared this way. But, what is appropriate for you really depends on what it is you're solving.