I am working a paper which offers an algorithmic improvement, which is measured in both the quality of the results, and the time it took to achieve them.
There are 3 recent (relatively) papers which deal with the exact same problem. One of them i am building upon, and my improvement (while can be used in other fields as well) is geared for it. The improvement is significant, and I have all the required tools in order to reproduce its exact results and prove it.
However, the 2 other Papers, which both (claim) to be better than the first, lack the required code in order to reproduce the results. I have tried contacting the authors, in more then one way, but no reply.
Also, according to paper #3 (which is the most recent one), paper #1 actually performs better than paper #2, A thing which contradicts paper #2.
I have an implementation of paper #2 (Not my own, or the original) which exhibits very poor results.
I am not sure how I can move on from here - Doing my own implementations for #2 and #3 is both very time consuming, and I could easily have a bug or two. And a reviewer can claim it was biased against them.
And obviously, If I wont benchmark against them, I could not claim to be The state-of-the-art solution for the problem (even if the improvement I show vs #paper 1 is bigger than they does).
Any advise would be great!