I am a beginning researcher in a particular real-world application space of machine learning, with papers published in the application space, but not in machine learning itself. I have identified a paper that is important to my program of research to replicate, have dedicated myself to its replication for a longer period of time than I wish to admit (even anonymously), and failed.
I am asking this question anonymously, and not identifying the particular application space of the paper (beyond the much larger "machine learning" domain) because the community of the application space is relatively small and I do not wish to identify myself or the authors of the paper I discuss below.
Some salient details:
I'm aware that some papers in machine learning are viciously difficult to replicate, even under the best of conditions. It is easy to find discussion forum comments such as "I tried to replicate X, spent months on it, got nowhere," or "I was only ever able to get Y to work by doing thing Z that wasn't in the paper."
Despite that, the machine learning community thrives on Github implementations by original authors, and Github re-implementations by students. This research group has to my knowledge never published any code from any publication, and while some re-implementations of various papers from this research group exist, this one has none.
It seems, at a minimum, certain key details are absent from the paper such as hyper-parameters, learning rates, and other customary details. Although in other respects the reported architecture is quite detailed, this does cause me to wonder if the model is unusually sensitive to the hyper-parameters, or if some other small detail has been left out.
A typical piece of advice given when I ask other researchers about this is to e-mail the original authors. I have done this, as politely as possible (making the explicit assumption that I'm just missing something and asking very specific questions, i.e., about the hyper-parameters directly, so as not to waste their time) several times, and gotten no response at all, even offering to post my own re-implementation on Github, to their credit. (I've sent enough e-mail that I fear one more will make me look like a stalker.)
Another typical piece of advice is, "Can you fix it and publish the new and improved version?" Believe me, I have tried, both minor variations on the paper, and updating it with more recent machine learning techniques, with no useful results. I have nothing publishable for my efforts.
Here are two final factors which complicate the matter even more in my mind (and which I think cause this not to be an exact duplicate of prior questions):
Although the lead author and their research group are well-regarded and well-published, this particular paper is published only on Arxiv There is no peer review, no conference or journal, and there is an "at your own risk," factor involved.
Despite this, the Arxiv publication is gathering citations from other refereed publications, although it is not obvious to me whether the citing works' authors have been able to (or even necessarily tried to) replicate the results, either.
What can one, and what should one do, when one is Nobody from Nowhere as far as a research community is concerned, but is beginning to suspect that a cited but not peer-reviewed paper cannot be replicated?
