4

I am performing research on a particular architecture in deep learning. Initially, the state-of-the-art research paper in Arxiv related to architecture is showing a score of (say) 9. Later the value is updated to 10 in the research paper. In the corresponding GitHub account, the code for the architecture is the one that is capable of giving 9 only and the authors did not update the code or other contents in the research paper. Anyone can able to get the value 9 if they execute on their system. The tickets are open and the authors didn't respond to any of the tickets that are asking for the code that is capable of giving the value 10.

It may be the reason that the research may be undergoing and authors are not willing to publish either the technique or their code.

Although the paper got citations, it is not published in any Peer review journal yet. The citations are, I think, for the architecture that is capable of generating 9 since no one can able to get 10.

Suppose I got the value 9.5, can I send it to any Peer review journal for Publication since the technique or code is not available in public and the authors are just claiming the value of 10?

Improve this question
3
  • 1
    Can I know the reason for the downvote?
    – hanugm
    Commented Oct 2, 2021 at 3:43
  • It's not an answer, but as a reviewer I don't really care whether someone reports 9, 9.5, or 10 -- for me, it's the same ballpark, with the result contingent on the current phase of the Moon and minor deviations in code & data. Thus, I'd position my approach as having some other merits rather than providing marginally better scores in some synthetic tests.
    – rg_software
    Commented Oct 2, 2021 at 6:15
  • 1
    Rather than publishing this paper, I feel it would be better if you have a preprint for it. If your technique is completely different from the one they have used, you could try expanding your study to a different dataset which they have not tested on.
    – Academic
    Commented Oct 2, 2021 at 7:48

1 Answer 1

Reset to default
7

If I understand correctly, this group has published an algorithm and claims to achieve a score of 10 in numerical experiments. The presumption will be that this result is correct, even if the publicly-available source code does not reproduce that number. Now it's true that one could write their own paper saying "this group's results are not reproducible; I did exactly what they said (using their own code!) and got a score of 9." But doing this might be inadvisable, since the other group will likely be able to defend their claim of 10, and then you will look bad for making a false accusation.

It certainly is annoying when groups publish buggy source code that is not aligned with their own results. But remember, most journals do not require source code at all. Even some major AI/ML results from Google use proprietary datasets, so no one can reproduce their results; we just have to trust them that the results would be reproducible if we had a similar dataset. This is really not so unusual in science; the same is true for results from large telescopes or particle colliders, for example.

Suppose I got the value 9.5, can I send it to any Peer review journal for Publication since the technique or code is not available in public and the authors are just claiming the value of 10?

You can send whatever you want to a journal. Certainly you can "make the case" that a reproducible 9.5 is better than an irreproducible 10. And perhaps your algorithm has other differences/advantages that make it interesting. Nonetheless, the presence of this previous paper that claims to get better results will be a major adverse factor when the reviewers decide whether to accept your paper.

Improve this answer
3
  • I am curious, how do mistakes (or even deception) get caught if everyone assumes that all claimed results are correct? It is understandable for the Google-scale experiments and proprietary datasets, since almost no one can even try to reproduce them, but for smaller scales, when code is unavailable, shouldn't it suffice to reimplement the methods described in the existing paper and compare with your own? This is what quite a few papers I have read seem to do. After all, the results will be comparable only if the settings (dataset, architecture, etc.) in both are identical.
    – GoodDeeds
    Commented Oct 2, 2021 at 9:29
  • @GoodDeeds Short answer: they commonly don't in AI/ML. The field is moving so quickly barely anyone has time for those; if the result is not particularly interesting/promising, it's going to be abandoned in an year tops, much to the demise of those seeking to reimplement it thinking it'd help with some SOTA-related part or with a non-mainstream dataset. The reason for this is that it's lot more industry-leaning rather than science mindset: one doesn't care whether the results are correct, only how much effort would it take to make it work for them and whether it's applicable to their problem
    – Lodinn
    Commented Oct 2, 2021 at 18:09
  • @GoodDeeds - yeah, if there is a significant claim (e.g., we set a new record on ImageNet), the sort of tests you describe will normally be done by the authors prior to publication, and there will be lots of interest in reproducing the results. But for less significant work (i.e., most papers), someone might personally conclude that the article is bogus, but due to the low attention any given paper receives, it is highly unlikely they would go through the process to formally challenge a paper.
    – cag51 ♦
    Commented Oct 3, 2021 at 7:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .