0

I am working a paper which offers an algorithmic improvement, which is measured in both the quality of the results, and the time it took to achieve them.

There are 3 recent (relatively) papers which deal with the exact same problem. One of them i am building upon, and my improvement (while can be used in other fields as well) is geared for it. The improvement is significant, and I have all the required tools in order to reproduce its exact results and prove it.

However, the 2 other Papers, which both (claim) to be better than the first, lack the required code in order to reproduce the results. I have tried contacting the authors, in more then one way, but no reply.

Also, according to paper #3 (which is the most recent one), paper #1 actually performs better than paper #2, A thing which contradicts paper #2.

I have an implementation of paper #2 (Not my own, or the original) which exhibits very poor results.

I am not sure how I can move on from here - Doing my own implementations for #2 and #3 is both very time consuming, and I could easily have a bug or two. And a reviewer can claim it was biased against them.

And obviously, If I wont benchmark against them, I could not claim to be The state-of-the-art solution for the problem (even if the improvement I show vs #paper 1 is bigger than they does).

Any advise would be great!

1 Answer 1

4

In this situation we always just say "we attempted to compare our result against XYX et al, but could not access a working implementation."

If a reviewer were to complain, we might say (to the reviewer, not in the paper) "in what way does #3 move the field forward if it cannot be used by anyone".

2
  • +1 Your results should stand on their own, not just in relation to others.
    – Buffy
    Jul 1, 2019 at 13:38
  • 1
    I would even add that, if there's no way to implement their code (either through replication or by contacting them), I question how much emphasis you place on their work, other than to mention that other attempts have been made but could not be replicated. (Yes, it may be a bit of a dig to say that you couldn't replicate their work, but that is the price of non-open science.) [Unless there is a compelling reason keep an algorithm proprietary, I have a very dim view of scientific publishings that withhold data / algorithms.]
    – Van
    Jul 1, 2019 at 13:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .