How to deal with potential data falsification?

Question

TL;DR

We cannot reproduce very good simulation results of our paper and the person who was responsible for writing the code and running the simulations does not have the code anymore and cannot explain how the results were obtained. How should we deal with potential data falsification?

Long version

I co-authored a paper that included some Monte Carlo simulations. One of the coauthors was responsible for the simulations, allegedly wrote the code and gave us the results of the simulations. Later on, I wanted to reuse the code for another project. He reluctantly sent me the code after a long delay. However, the code did not generate the results that we reported in the paper. He said that he lost the code that he used to generate the results of the paper. The code that he sent me is some code that he used at some point of writing the code for the paper, but might not be the final one. He could not explain how the results of the paper were obtained.

We were also working on another project with the same person. He was also responsible for simulations of that project. He again lost the code and could not explain how the results were obtained. The paper of the second project was rejected by the journal and we are going to be able to rerun the simulations and report the correct results.

I suspect that this person just falsified the results.

We are thinking about removing this person from the project and removing this person from the list of authors when we resubmit that paper. Even though the person cannot explain how the results were obtain, there is still a chance that he somehow made some mistake that led to very good results. However, I think it is unlikely. Removing this person from the list of the authors would probably affect his career in a bad way so I would not want to do that without enough evidence. However, I do not think that we can continue working with this person because he lost our trust.

What should we do? How should we deal with this potential data falsification by a coauthor?

Answer 1

In science, a lack of reproducibility leaves you all on the hook. But, I would probably recommend getting a third expert in Monte Carlo to review the code. I'll assume this is you. First, it can be the case that a small change in code makes big differences, permutation testing for instance can do this. But, you can run the simulation lots of times, look at the results, and find what quintile the original results are in. This corresponds to a p value for falsification, or as I will call it: a possible mistake. Nonetheless, Monte Carlo is not very complicated, and in the paper regardless you will have to explain how the answers were obtained. If you cannot do this, you cannot push paper into community. So, inspect, conclude, rework. For their sake, I hope falsification is not the case. That's a very very ugly way to go about research, and it will crop up.

Answer 2

This is a fairly tricky issue, so let us start with the most straightforward aspects and then move into the complications.

(1) Reproducibility is key, no two ways about it. With any experiement or model, within reasonable error bounds, your results should be reproducible. If there is a co-author who does the simulation, (s)he must be made aware of this as a precondition to submission. If there is scatter in the results, (s)he should be able to explain it. If not, don't use those results. Either way, it's your responsibility as lead author (I'm assuming you're it) to check this aspect. How you check it depends on interpersonal factors- how trustworthy this person is, can you insist on seeing results live, etc.

I would not recommend asking for the code and running it yourself- however simple the code, the writer has put effort into it, so giving it away may seem unfair. Even if you are co-authors here, you may not be later, and you could use this for work without this person.

(2) Code foibles: Code should be clean, easy to understand and have good documentation. Sadly, as is frequently discussed here, most research code does not have these qualities. So there is a chance that the original writer of the code knows some shortcomings and the solutions to those, which may not be clear to you. (S)he may apply these corrections to the results subsequently. This is all not ideal, but happens frequently if people aren't experienced programmers.

(3) Point (2) opens up the possibility of falsification. But only if this co-author knows the expected outcome and details of experiment. We practice keeping everything insulated while generating data/results- person using instrument A knows only what's essential for A, and likewise for the other experiments and models. Once all the results are obtained, we sit together and see what to make of it. It's a simple bias blocker, and it may not always be possible in this form. But the principle is worth following.

At present, you can implement only (1) and (2), while (3) is for the future.