18

In computer science, we usually implement our work into algorithms and codes. So in order to compare our work to others, we also need to implement their approaches as well.

But the problem is that usually there are recently published works in each area which have used complicated mathematical methods and claimed to beat other previous methods. But some of them are difficult to implement by just reading the papers; or they have not mentioned all the necessary details in the paper for a precise implementation of their work and neither have they make their code available.

How should I compare my work to these most recent algorithms for which the code is not available?

  • 2
    You can compare at the theoretical level, rather than at the practical (implementation) level. – user2768 Apr 30 '18 at 9:12
  • 19
    No published working implementation = ignore it. It’s bad science anyway, since their claims from the paper aren’t verifiable. – Konrad Rudolph Apr 30 '18 at 17:20
23

Asking the authors for the code is of course always an option, but from my personal experience this almost never works (there is a reason why they did not put it online in first place).

I don't know which area in computer science you are from but normally there is some quantitative measure to determine which method is better and those measures are somehow in the paper. Those can be things like run time or precision-recall curves on certain data sets.

If you want to compare on the same dataset, you can ask the authors for the numbers / plots themselves and then plot your own curves together with them.

If you want to compare on a different data set, you can ask the authors to run the code for you. In my experience, this works more often because many code is not published because its messy and needs a lot of parameter tuning - so it is unlikely someone else will produce good results with it.

If you have Bachelor/Master students to supervise, you can give the implementation as a project to them (although you need to give them enough time to do it).

In my personal opinion, it is not worth the effort to implement a method which is not described in enough detail anywhere. It will probably cost you an unreasonable amount of time without even reproducing the quality shown in the paper (for various reasons).

  • 3
    "many code is not published because its messy and needs a lot of parameter tuning - so it is unlikely someone else will produce good results with it." And there we have the difference between a CompSci person and a Software Developer. Although admittedly there is often some overlap.... but at the same time, not nearly enough overlap. – ivanivan Apr 30 '18 at 11:19
  • 1
    I know it is possible to compare the results to the numbers given in their papers, but you cannot have a fair comparison unless you are sure using same experimental setup, for example, the type of cross-validation or the pre-processings they used regarding the data. – Babak Apr 30 '18 at 11:43
  • 20
    To quantify "this almost never works": There is a study by Collberg et al., in which the authors of 263 papers wered asked to provide their source code. The authors responded by providing their source code in 87 cases (33% of all cases). – lighthouse keeper Apr 30 '18 at 15:16
  • 6
    It seems like if the authors can't/won't provide source code and actively make it difficult or impossible to verify/reproduce their results, this should be cause for making a complaint to the journal that published the result. – R.. Apr 30 '18 at 18:53
  • 10
    "If you want to compare on a different data set, you can ask the authors to run the code for you." At which point your paper contains results based on experiments you didn't perform using code you don't have. What happens when someone contacts you trying to reproduce your results? – Ray Apr 30 '18 at 20:23
1

If you publish your code, and it produces better quantifiable results than any other published piece of code, then you can claim it is the best. For anyone to dispute that, they would need to show better quantifiable results. In science, we need to be able to test theories and in computer science, that means testing algorithms. Peer review applies to computer science as well.

1

What's wrong with being honest?

We compare our algorithm to A and to B. We would have liked to compare also with X, Y and Z, but code is not available for them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.