I am working a paper which offers an algorithmic improvement, which is measured in both the quality of the results, and the time it took to achieve them.

There are 3 recent (relatively) papers which deal with the exact same problem. One of them i am building upon, and my improvement (while can be used in other fields as well) is geared for it. The improvement is significant, and I have all the required tools in order to reproduce its exact results and prove it.

However, the 2 other Papers, which both (claim) to be better than the first, lack the required code in order to reproduce the results. I have tried contacting the authors, in more then one way, but no reply.

Also, according to paper #3 (which is the most recent one), paper #1 actually performs better than paper #2, A thing which contradicts paper #2.

I have an implementation of paper #2 (Not my own, or the original) which exhibits very poor results.

I am not sure how I can move on from here - Doing my own implementations for #2 and #3 is both very time consuming, and I could easily have a bug or two. And a reviewer can claim it was biased against them.

And obviously, If I wont benchmark against them, I could not claim to be The state-of-the-art solution for the problem (even if the improvement I show vs #paper 1 is bigger than they does).

Any advise would be great!


In this situation we always just say "we attempted to compare our result against XYX et al, but could not access a working implementation."

If a reviewer were to complain, we might say (to the reviewer, not in the paper) "in what way does #3 move the field forward if it cannot be used by anyone".

  • +1 Your results should stand on their own, not just in relation to others.
    – Buffy
    Jul 1 '19 at 13:38
  • 1
    I would even add that, if there's no way to implement their code (either through replication or by contacting them), I question how much emphasis you place on their work, other than to mention that other attempts have been made but could not be replicated. (Yes, it may be a bit of a dig to say that you couldn't replicate their work, but that is the price of non-open science.) [Unless there is a compelling reason keep an algorithm proprietary, I have a very dim view of scientific publishings that withhold data / algorithms.]
    – Van
    Jul 1 '19 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.