I plan to submit an article to a top tier conference in the field of computer science. However, I have certain doubts about the right procedure to compare with state-of-the-art methods in my field.

I want to claim that the approach we propose is better than the state-of-the-art method. To prove this, I have re-implemented the state-of-art and tested it on a benchmarked dataset of my own creation. The method of our proposal seems to outperform the state-of-the-art.

However, the authors of the state-of-the-art paper had neither open-sourced their code nor tested it on a publicly available, benchmarked dataset. So, it is proving to be pretty much impossible to confidently claim that our re-implementation of the state-of-the-art is an accurate replica of what was proposed/implemented in the original paper.

How do I go about highlighting this dilemma in the article I am about to submit, without sounding too vague? I am ready to share a detailed pseudo-code of the re-implementation as supplementary material to the submission. I am also about to make the benchmarked dataset (that we created) public.

However, I would like to ask: what is the standard procedure for handling such doubts? (I am sure I am not the first author to face such a confusion).

  • 2
    possible duplicate: academia.stackexchange.com/questions/138490/… – Ian Sudbery Nov 9 '19 at 16:12
  • Why are you only prepared to share pseudo-code of your reimplementation. Whats wrong with sharing the code? – Ian Sudbery Nov 9 '19 at 16:13
  • @IanSudbery: I am already sharing the code of my own implementation. The re-implementation is done on top of a very large framework, and to render it readable is quite a task. I can also share screenshots of the actual code, but sharing the entire framework which is compilable and replicable is a bit too much for me at the moment. All this for one of many comparisons that occupies two rows of one table in the entire paper! – metsburg Nov 9 '19 at 20:53
  • I do get the answer in the other thread that you shared. – metsburg Nov 9 '19 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.