How to deal with non-reproducible research?

Question

For context, I am a PhD student in bioinformatics, working partly in development of new statistical models and partly in applying models to data.

I recently came across a paper which describes an interesting approach to clustering data. This method would have direct use for data I was working with at the time. The model itself was also interesting, and I am interested in using ideas from their paper to solve similar problems in my field.

I tried to use their published modelling software to analyse my data. It crashed my R session. For context, errors in R code rarely cause the session itself to crash; usually they result in an error message, which one may easily find the source of.

Unsure of whether this was my mistake (due to a coding error or data misspecification), I retrieved the data from their publication and attempted to run the code which they state produces the outputs in their paper. This lead to the same issue. I flagged this issue on the software repository, and another user mentioned having the same issue. I have replicated the issue in multiple independent environments using different software versions. I have also informed them that I have narrowed the error down to one or more lines of code, though I cannot exactly explain why it occurs (it would be too time-consuming to debug as I am not familiar with their code).

The authors initially did not respond to the issues raised on the repository, and only did so after I emailed the corresponding authors several weeks later. They have only stated that the software works on their system and suggested that I try different software versions, which I have now done with no change in outcome.

My initial goal was to ensure the issue was reproducible, and to raise it to the authors' attention. If the issue remains unresolved, and the authors largely absent if not uncooperative, how should one deal with this? I do not feel that this amounts to research misconduct, however I also feel unable to trust the results in their paper, given that the code which should replicate their results does not function in any respect.

Answer 1

As an R package author myself I can only say that you did the best you could. I'm grateful if users flag issues like this to me, and I do my best to correct errors in my code (which do happen) responding to such requests.

I'm not sure how insistent you were with the authors; they may have said "try a different version" to fob you off, but may still react if you tell them "this-and-that doesn't work either". It is easy to get lost in changing versions of dependent packages etc., so occasionally an analysis initially gives a certain result and then something like an update of a required external package happens that breaks it. However I'd normally expect the authors to collaborate getting at the bottom of the issue.

Assuming that the authors remain uncollaborative you are right to not trust their results and not use their method. To what extent you want to call them out is up to you really. Maybe their results were fine originally and they just can't find the time to investigate the present error. Who knows? But surely if others discuss the package feel free to share your experience. There's too much stuff out there that doesn't work! I'm sure this is an everyday story, unfortunately.

Answer 2

R crashing

errors in R code rarely cause the session itself to crash; usually they result in an error message, which one may easily find the source of

yes. There are a few situations where session crashes are more probable:

• with compiled code and
• libraries installed on the system (as opposed to R packages) which are incompatible across versions/OS boundaries
• particularly libraries that are "close" to the OS (I've had trouble in the past with interactive graphics across OS boundaries Linux vs. MacOS vs. Windows)

Thus, if the lines in question can (easily) be replaced by equivalent (though maybe slower or more memory conuming) R code, the crash may be avoided and the scientific merits of the model may be evaluated.
We may have an example here of the famous "premature optimization is the root of all evil".

---

Amount of cooperation to be expected?

As author and maintainer of scientific/academic software packages I certainly appreciate bug reports etc. I even more appreciate bug fixes, btw.

However, I also have only limited resources available to maintain such software, and the longer back the publication was, typically the less resources (e.g. new employer is not willing for their staff to maintain projects at former universities) are standing agaist increased maintenance requirements. (Probably less of a problem here as the paper is quite recent compared to some code that was developed years and years back)

Please also have a look at Why do many talented scientists write horrible software?

The authors may be seeing the publication of the code as them being extremely nice in giving the public not only the abstract description of the model but even an example of an implementation.
In fact, they may even have had an uphill fight to get permission to publish their code under an open license (I've been in the situation of not being permitted to do this by an academic research institution.)

Of course it would be much nicer if they had written more rugged code in the first place or would now help you debug. But you don't have any kind of hard right to their time.
I'm chemist and we sometimes have lab procedures that are difficult to describe in such detail that another lab with somewhat different conditions is [easily] able to reproduce them. Thus, people visit other labs to learn their techniques - this means a whole lot of effort for both sides and IMHO that needs to be appreciated. Similarly, I appreciate if a package maintainer does put in the time to deal with my bug report. And in consequence, I try to make it as easy as possible for them.

There may be a mismatch in software development ability/possibilities here: guessing from the crashes that their code isn't at the highest level of robustness and that you didn't find them using continuous integration etc. vs. you providing them with a Singularity container: they may not know how to use that or may not have the possibility to use it or may not be willing to put in the time to get that particular virtual machine up and running.

I may add that in many places (including universities and research institutes) there is in addition daunting burocracy to get the IT department to install further software on their machines (and they are the only ones with installation rights). I teach for the carpentries and it is not unusual to find course participants do not have the necessary software for this reason - even if they are officially sent to the course.

You may get further if you ask them whether they do have the ressources to resolve the issue with your help and how you can help resolving the issue.

---

Misconduct?

I do not feel that this amounts to research misconduct

No, it isn't misconduct - it's just that the situation could be nicer.

It would be academic misconduct if the code at the time of writing and submitting their publication on the authors' machines had not produced the results described (incl. crashing and not producing any results).

In contrast,

• writing code that is not portable or
• fragile in the sense that it is quite likely to break when its surroundings (OS, interpreter, dependencies) evolve and
• abandonware

are not scientific misconduct even if it is not "best programming practice".

Trust in their results?

That's a difficult one.

• On the one hand, with the possible exception of computer science, the scientific ability of the authors may be quite uncorrelated to their software development abilities (see linked answers above)
• On the other hand, if their software doesn't calculate what they think (claim) it does, then also the scientific content may be compromised.

For the case in question:

• I tend to think that crashes (or stopping with an error message) are often relatively harmless in terms of scientific integrity of the paper. Short of blatant misconduct (claiming results that weren't obtained) this points to the code not being robust/maintained in an evolving environment (or data/formatting subteties) and not necessarily an incorrect implementation on their system.

• I'd be more concerned about the lack of unit tests confirming the results of calculations that actually run through: statistics offers lots of possibilities to have logical error which lead to wrong but often even plausible numbers. That's what is scary to me...

What to do

• Be extremely nice to them.
  In many fields, it is not (yet) standard to publish code at all. They did you a favor in giving you an implementation that presumably works on their machine. In fact, as long as it takes you less time to fix & test their package than to write & test your own implementation of their paper, you have a net gain!

  I don't want to insinuate that you are not nice to them.

  However, my experience of some 10 years as (more or less active according to the ressources described above) maintainer of an R package is that the majority of help requests is answered with RTFM and many bug reports do not provide a minimal working example that I can reproduce and there are a few insistent and obnoxious help requests that suck up your time like a black hole because they try to offload the time they should put into learning R onto you package maintainer. While I try hard to treat all requests in a professional, friendly and timely manner, I also sometimes fail in that (most often fail is: timely). (And I have to say that those bad experiences are offset by also receiving extremely well written issue reports, sometimes with immediately usable pull requests and finding online contributors whom I'd otherwise never have known and even occasional "thank you for this package" emails. But there's a whole lot of truth to bad interactions producing far more impression than good interactions.)
  You may just have had the bad luck to happen to be the one after a couple of toxic requests and/or may have inadvertendly triggered such an alarm with the authors.
  Again while that is not the ideal interaction, you best bet to get the issue resolved is to be so supernice as to convince the authors that not all interactions on the internet are bad... Gives you good karma, too.

• Encourage collaboration and make it easy for them: do put in the time to dig into those lines for debugging. Or write an R workaround and ask them to be so kind as to review your suggested changes.

• In order to get the code up and running, ask them whether they can provide you a sessionInfo () of their system and the versions of the relevant underlying libraries. The logical step after you trying several configurations that didn't work is to try and get a reproduction of "their" system to work.

• Of course, it is completely fine if you decide to stop sinking time into a not-so-well designed package and either

  • write your own implementation of the model described in their paper, or even
  • completely abandon that model if you distrust the science behind it.