11

TL;DR

We cannot reproduce very good simulation results of our paper and the person who was responsible for writing the code and running the simulations does not have the code anymore and cannot explain how the results were obtained. How should we deal with potential data falsification?

Long version

I co-authored a paper that included some Monte Carlo simulations. One of the coauthors was responsible for the simulations, allegedly wrote the code and gave us the results of the simulations. Later on, I wanted to reuse the code for another project. He reluctantly sent me the code after a long delay. However, the code did not generate the results that we reported in the paper. He said that he lost the code that he used to generate the results of the paper. The code that he sent me is some code that he used at some point of writing the code for the paper, but might not be the final one. He could not explain how the results of the paper were obtained.

We were also working on another project with the same person. He was also responsible for simulations of that project. He again lost the code and could not explain how the results were obtained. The paper of the second project was rejected by the journal and we are going to be able to rerun the simulations and report the correct results.

I suspect that this person just falsified the results.

We are thinking about removing this person from the project and removing this person from the list of authors when we resubmit that paper. Even though the person cannot explain how the results were obtain, there is still a chance that he somehow made some mistake that led to very good results. However, I think it is unlikely. Removing this person from the list of the authors would probably affect his career in a bad way so I would not want to do that without enough evidence. However, I do not think that we can continue working with this person because he lost our trust.

What should we do? How should we deal with this potential data falsification by a coauthor?

  • 1
    Its possible also that the other person's research is heavily based on those codes and do not want to share their code for fear of you removing them from the project while using their code. I learned that researchers are often very very protective of their code. – Ander Biguri May 22 '18 at 10:03
  • 1
    Have you told all this to the other person? Not the falsification part, but the fact that if you want to add those experiments in the code you need to be able to reproduce them and you are not being able. In a polite way ask for help, and tell them that you need the code for reproduciblility. If they are in fear of you copying the code, ask them to release the code privately with some license that doesnt allow anyone to use/modify the code, that way they can be reassured that you wont put them aside. – Ander Biguri May 22 '18 at 10:05
  • @AnderBiguri Thanks for the response! The code is relatively simple. I do not think that the coauthor would lie and hide his code just to protect it. I wrote another code from scratch that does the simulations of the paper, but I cannot reproduce the results of the paper (I get worse results). I told the coauthor that I cannot reproduce his results and asked if he can see where the problem is. – Cm7F7Bb May 22 '18 at 10:18
  • 2
    Then you need to rewrite the paper, and tell the other person that this happens. MC simulations are very random base by definition, but you can make randomness reproducible in computers by setting random seeds. Ask this to the other person. Just in generaly be very wary of calling someone out for cheating without proof. – Ander Biguri May 22 '18 at 10:44
  • 1
    Seems that your work is about performance. Have you used the same computer setup? The same compiler and the same compilation flags? For some codes optimization flags increase dramatically the efficiency. @Ander MC simulations are random, but you are supposed to have the same result, even using different seeds, if the number of simulations is statistically relevant. – The Doctor May 22 '18 at 12:25
1

In science, a lack of reproducibility leaves you all on the hook. But, I would probably recommend getting a third expert in Monte Carlo to review the code. I'll assume this is you. First, it can be the case that a small change in code makes big differences, permutation testing for instance can do this. But, you can run the simulation lots of times, look at the results, and find what quintile the original results are in. This corresponds to a p value for falsification, or as I will call it: a possible mistake. Nonetheless, Monte Carlo is not very complicated, and in the paper regardless you will have to explain how the answers were obtained. If you cannot do this, you cannot push paper into community. So, inspect, conclude, rework. For their sake, I hope falsification is not the case. That's a very very ugly way to go about research, and it will crop up.

0

This is a fairly tricky issue, so let us start with the most straightforward aspects and then move into the complications.

(1) Reproducibility is key, no two ways about it. With any experiement or model, within reasonable error bounds, your results should be reproducible. If there is a co-author who does the simulation, (s)he must be made aware of this as a precondition to submission. If there is scatter in the results, (s)he should be able to explain it. If not, don't use those results. Either way, it's your responsibility as lead author (I'm assuming you're it) to check this aspect. How you check it depends on interpersonal factors- how trustworthy this person is, can you insist on seeing results live, etc.

I would not recommend asking for the code and running it yourself- however simple the code, the writer has put effort into it, so giving it away may seem unfair. Even if you are co-authors here, you may not be later, and you could use this for work without this person.

(2) Code foibles: Code should be clean, easy to understand and have good documentation. Sadly, as is frequently discussed here, most research code does not have these qualities. So there is a chance that the original writer of the code knows some shortcomings and the solutions to those, which may not be clear to you. (S)he may apply these corrections to the results subsequently. This is all not ideal, but happens frequently if people aren't experienced programmers.

(3) Point (2) opens up the possibility of falsification. But only if this co-author knows the expected outcome and details of experiment. We practice keeping everything insulated while generating data/results- person using instrument A knows only what's essential for A, and likewise for the other experiments and models. Once all the results are obtained, we sit together and see what to make of it. It's a simple bias blocker, and it may not always be possible in this form. But the principle is worth following.

At present, you can implement only (1) and (2), while (3) is for the future.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.