0

I am a beginning researcher in a particular real-world application space of machine learning, with papers published in the application space, but not in machine learning itself. I have identified a paper that is important to my program of research to replicate, have dedicated myself to its replication for a longer period of time than I wish to admit (even anonymously), and failed.

I am asking this question anonymously, and not identifying the particular application space of the paper (beyond the much larger "machine learning" domain) because the community of the application space is relatively small and I do not wish to identify myself or the authors of the paper I discuss below.

Some salient details:

  • I'm aware that some papers in machine learning are viciously difficult to replicate, even under the best of conditions. It is easy to find discussion forum comments such as "I tried to replicate X, spent months on it, got nowhere," or "I was only ever able to get Y to work by doing thing Z that wasn't in the paper."

  • Despite that, the machine learning community thrives on Github implementations by original authors, and Github re-implementations by students. This research group has to my knowledge never published any code from any publication, and while some re-implementations of various papers from this research group exist, this one has none.

  • It seems, at a minimum, certain key details are absent from the paper such as hyper-parameters, learning rates, and other customary details. Although in other respects the reported architecture is quite detailed, this does cause me to wonder if the model is unusually sensitive to the hyper-parameters, or if some other small detail has been left out.

  • A typical piece of advice given when I ask other researchers about this is to e-mail the original authors. I have done this, as politely as possible (making the explicit assumption that I'm just missing something and asking very specific questions, i.e., about the hyper-parameters directly, so as not to waste their time) several times, and gotten no response at all, even offering to post my own re-implementation on Github, to their credit. (I've sent enough e-mail that I fear one more will make me look like a stalker.)

  • Another typical piece of advice is, "Can you fix it and publish the new and improved version?" Believe me, I have tried, both minor variations on the paper, and updating it with more recent machine learning techniques, with no useful results. I have nothing publishable for my efforts.

Here are two final factors which complicate the matter even more in my mind (and which I think cause this not to be an exact duplicate of prior questions):

  • Although the lead author and their research group are well-regarded and well-published, this particular paper is published only on Arxiv There is no peer review, no conference or journal, and there is an "at your own risk," factor involved.

  • Despite this, the Arxiv publication is gathering citations from other refereed publications, although it is not obvious to me whether the citing works' authors have been able to (or even necessarily tried to) replicate the results, either.

What can one, and what should one do, when one is Nobody from Nowhere as far as a research community is concerned, but is beginning to suspect that a cited but not peer-reviewed paper cannot be replicated?

  • 1
    Do your own results seem to refute the paper or support it generally if not specifically? – Buffy Jan 30 at 11:57
  • All but the last major stage of the approach works. There is a technical name for the practical failure it exhibits, which is barely addressed in passing, and not by its technical name. But the failure could be mine generally; or lack of parameters not presented (but necessary) in the paper. – Anonymous Jan 31 at 4:28
1

I would suggest focusing far, far less on this non-peer-reviewed paper.

Publish the results you have. Use methodology that is as sound as you can achieve with the data and tools that you have. Cite previous work that agrees and disagrees with what you find. Contextualize your findings in an understanding informed by an entire body of work, not just individual papers.

The only time it makes good strategic sense to focus on replication of a particular result is if that result is influential and relied upon in practical use. A discerning reader of the literature will approach any single finding with appropriate skepticism.

|improve this answer|||||
0

What can one, and what should one do, when one is Nobody from Nowhere as far as a research community is concerned, but is beginning to suspect that a cited but not peer-reviewed paper cannot be replicated?

You could publish what you've discovered. One criticism you may receive is: The original work isn't published, so your results aren't interesting. You could defend against that, e.g., by explaining the interest others have in the original work (as illustrated by citations).

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.