18

In computer science, we usually implement our work into algorithms and codes. So in order to compare our work to others, we also need to implement their approaches as well.

But the problem is that usually there are recently published works in each area which have used complicated mathematical methods and claimed to beat other previous methods. But some of them are difficult to implement by just reading the papers; or they have not mentioned all the necessary details in the paper for a precise implementation of their work and neither have they make their code available.

How should I compare my work to these most recent algorithms for which the code is not available?

Improve this question
2
  • 2
    You can compare at the theoretical level, rather than at the practical (implementation) level.
    – user2768
    Commented Apr 30, 2018 at 9:12
  • 19
    No published working implementation = ignore it. It’s bad science anyway, since their claims from the paper aren’t verifiable.
    – Konrad Rudolph
    Commented Apr 30, 2018 at 17:20

3 Answers 3

Reset to default
25

Asking the authors for the code is of course always an option, but from my personal experience this almost never works (there is a reason why they did not put it online in first place).

I don't know which area in computer science you are from but normally there is some quantitative measure to determine which method is better and those measures are somehow in the paper. Those can be things like run time or precision-recall curves on certain data sets.

If you want to compare on the same dataset, you can ask the authors for the numbers / plots themselves and then plot your own curves together with them.

If you want to compare on a different data set, you can ask the authors to run the code for you. In my experience, this works more often because many code is not published because its messy and needs a lot of parameter tuning - so it is unlikely someone else will produce good results with it.

If you have Bachelor/Master students to supervise, you can give the implementation as a project to them (although you need to give them enough time to do it).

In my personal opinion, it is not worth the effort to implement a method which is not described in enough detail anywhere. It will probably cost you an unreasonable amount of time without even reproducing the quality shown in the paper (for various reasons).

Improve this answer
8
  • 3
    "many code is not published because its messy and needs a lot of parameter tuning - so it is unlikely someone else will produce good results with it." And there we have the difference between a CompSci person and a Software Developer. Although admittedly there is often some overlap.... but at the same time, not nearly enough overlap.
    – ivanivan
    Commented Apr 30, 2018 at 11:19
  • 1
    I know it is possible to compare the results to the numbers given in their papers, but you cannot have a fair comparison unless you are sure using same experimental setup, for example, the type of cross-validation or the pre-processings they used regarding the data.
    – Bob
    Commented Apr 30, 2018 at 11:43
  • 21
    To quantify "this almost never works": There is a study by Collberg et al., in which the authors of 263 papers wered asked to provide their source code. The authors responded by providing their source code in 87 cases (33% of all cases).
    – lighthouse keeper
    Commented Apr 30, 2018 at 15:16
  • 7
    It seems like if the authors can't/won't provide source code and actively make it difficult or impossible to verify/reproduce their results, this should be cause for making a complaint to the journal that published the result.
    – R.. GitHub STOP HELPING ICE
    Commented Apr 30, 2018 at 18:53
  • 12
    "If you want to compare on a different data set, you can ask the authors to run the code for you." At which point your paper contains results based on experiments you didn't perform using code you don't have. What happens when someone contacts you trying to reproduce your results?
    – Ray
    Commented Apr 30, 2018 at 20:23
3

What's wrong with being honest?

We compare our algorithm to A and to B. We would have liked to compare also with X, Y and Z, but code is not available for them.

Improve this answer
2
  • At least in our field it is frowned upon and gets rejected with "should have compared to methods X, Y, Z." or "lack of comparison". It's incredibly frustrating, but even stating that there is no code available does not seem to sway every reviewer.
    – dennlinger
    Commented Dec 16, 2020 at 10:18
  • I would also add that you have reached out to the authors but were not able to obtain the code to evaluate performance. This helps against the reviewer rebuttals. If you've reached out to the authors, and the code is not publicly available, most reviewers will not generally expect you to implement it by hand.
    – Pro Q
    Commented Sep 20, 2023 at 17:37
1

If you publish your code, and it produces better quantifiable results than any other published piece of code, then you can claim it is the best. For anyone to dispute that, they would need to show better quantifiable results. In science, we need to be able to test theories and in computer science, that means testing algorithms. Peer review applies to computer science as well.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .