I found published results of my mentor to be wrong. How to tell her?

Question

For the last few days, I have been trying to reproduce things in a published paper (in quite a reputed journal) from my field. My postdoc mentor is the last author of the paper and the main work was carried out by her collaborators from a different institute and a first author is finishing his Ph.D.

When I saw the paper, I immediately got skeptical because they had missed some obvious things, so I rewrote all the code from scratch and, after a week's extensive checking and cross-checking, confirmed that their results are completely wrong. I told two of my trusted colleagues and they asked me to tell my mentor this. However, I am quite uncomfortable and confused about this.

The student who is the first author has based a big part of his later work on these results. This means that the later work by the student might also be completely wrong. My question is how to proceed from here? Can I ask my mentor to retract their paper? Or should I ask her permission to write a comment about the paper? If I do either of these (or something else which reveals that those results are wrong), would I be considered guilty of destroying the thesis of the first author guy?

I really appreciate any type of help in this matter. Thanks in advance.

Edit: All the answers and comments here are very helpful and I thank you all for them. However, I think I was slightly vague while asking. I very well understand that I could be certainly wrong. But the question is what do I do if I am not? In that case, what should be done? (As things stand, they have surely got at least one plot wrong because it directly goes against a famous fundamental theorem. I got this confirmed from a few people in the field.)

Answer 1

As a matter of fact, I believe there is no such thing as guilt or ruin for other people's work when it comes to the academic aspect of the knowledge. Actually, if you do not share this with your mentor in order to be fixed, you may ruin many other further research...

I suggest you do this respectfully and carefully. You can go to your mentor and talk about this problem. You can show her your code and your understanding from the paper, then confirm whether or not you are right. Let her decide what she can do about this. There is also a slight chance you have made a technical mistake while investigating their work, so keep this in mind when you discuss the issue.

Answer 2

I suggest you treat this as a learning process. After all, you have spent on this only a week, while the authors have spent significantly more, so there is a chance you are wrong.

So, with this working theory in mind, present your results to your mentor and discuss with her what you think is wrong with their work and why and ask her to explain what you could be missing.

After the discussion you should, ideally, either realise why you were wrong; convince them of what they did wrong; or all of you agree to disagree on the correctness of the results. In the second and third scenario, your mentor should be able to suggest some path.

I do not think you should be asking them to retract the paper unless they have shown gross negligence in the research process or outright cheated; it should be up to them.

Answer 3

I think there is a simple answer. From what you wrote it sounds like you rather "could not reproduce the results" than "proved that the results are wrong". While the first is often an indication for the second, this need not be the case (especially since you "rewrote the entire code").

Now focus on a solution instead of the problem. What you should do is: Approach your mentor with "I tried to reproduce the findings of the paper in this and that way and ended up with different conclusions." From there on the story may evolve differently: It may be that there is a misunderstanding about the "missed obvious things", it may still be bugs in your code, it may that the paper was ambiguous about some things, also it may be that there is something wrong...

In this discussion be sure to be open minded and free of prejudice. As an example: When somebody tells me something in a scientific discussion that I think is wrong or bogus, I get a much better and helpful response when I say "Sorry, I did not understand that, can you say it differently?" than "That's nonsense."

Answer 4

I would recommend that instead of talking directly about the flaws of their work, you take an approach of explaining what you did.

Here's an example:

Hello Ms Mentoress, I have a question about that paper you published with co-authors A, B, and C a while ago. So I was trying to reproduce the results with code I've written on my own and I couldn't get it to work. Can you have a look and help me with this?

There are several reasons for using this approach:

1. You are not blaming anyone for anything, and you are not being aggressive. Retractions and the likes are not under discussion now, and they shouldn't be until the mistake is understood, by all parties involved.

2. There is a small possibility that you are wrong and not them. This will save you the embarrassment of blaming someone for being wrong while in fact they were right all along.

3. In the case that you are correct, your mentor will realise on her own that she was in fact wrong. She will hold you in high esteem for this, because in her mind you are now a scientist who can rigorously test other people's work, which in this case happened to be hers. This will build bridges instead of burning them

4. In the case this results in a comment or discussion paper, there's a good chance you will be co-author with that group of people who will most likely try to correct themselves. As these are probably people with some reputation who have published in high ranking journals, this will put you together with people you want to be in good relations with, not against them.

Answer 5

From personal experience, I know that it may be very difficult to replicate results. I would recommend first asking yourself the following questions: Can I explain in a sentence or two what drives your mentor's results? Can I do the correction that gets results I believe to be correct? Are these results central to the paper?

Allow me to elaborate: You need to be able to say "You get result Y only because you did X. But X is incorrect methodology because Z. When I try to replicate your result, instead of X, I do X', and then your result Y goes away. This is important because your result Y is central to the paper."

If you can fill in the variables in these sentences, then it may be time to talk to the mentor. It is then a matter of being diplomatic. Make sure that they understand that your intention to discuss these with your mentor only, and absolutely no third party. (Don't say this directly, but let this be your approach and let that show.) I would also recommend de-personalizing the issue in grammar: Say "these results" rather than "your results".

Finally, should you even talk about this? The decision is up to you and there is no clear answer; but it is a function of the answers to the following questions: How many years have you known your mentor? How good are you at being a diplomat? How insecure is your mentor as a person? What do you have to gain or lose in case of go / no go?

Answer 6

Right now, you don't know that your mentor's paper is "wrong." What you do know is that your results differ from hers.

Because she is your mentor, I would approach her in this spirit, and say, "I was trying to replicate your paper, and came up with entirely different results. One of us is doing something wrong, and it's probably me. Could you explain why your results are so different from mine?

It's possible that she'll point out some error in your logic or code, and you'll be saved the embarrassment of making a false accusation. It's also possible that you are correct, and by treading softly, you will keep her on your side. If you are right, you will give her the chance be a "hero" by contacting the others involved with the paper and suggesting that it needs to be corrected. Inwardly, she will be grateful to you for saving her from embarrassment.

Answer 7

Can I ask my mentor to retract their paper?

You can ask her anything. But you shouldn't. Don't tell her what to do; rather, tell her what you have found. If the paper is wrong, you should assume that the authors will behave with integrity and take the appropriate action: they almost always will.

A retraction is absolutely the last step, which will only happen if the paper really is wrong and it can't be corrected. It's not something you discuss the first time you mention that there seems to be something wrong: that would be a bit like beginning a conversation with, "I can't find my phone. Have you seen it? The maximum penalty for theft is ten years in jail."

Or should I ask her permission to write a comment about the paper?

No. On a technical level, you don't need anybody's permission to write and publish about their work. However, if you do end up publishing something saying "X is wrong", it's polite to let the authors of that work know in advance. It can also save a lot of time, because it means that there are more people working on the fix (you and them) and some of them are the world's greatest experts on that particular thing (them: even if they're the world's greatest experts on a wrong thing, they know much more about what they did than you do).

But you're not ready to write anything, yet. Talk to your mentor about what you've found and try to come up with a solution together. The only reason to continue without her is if she refuses to cooperate. In that case, because you're at an early stage of your career, I'd strongly recommend that you find somebody more senior to work with, since writing "X is wrong" papers is rather delicate.

If I do either of these (or something else which reveals that those results are wrong), would I be considered guilty of destroying the thesis of the first author guy?

No, not at all. Just as you're not guilty of destroying somebody's freedom if you call the police when you see them committing a crime. If people produce work that is incorrect, that is their fault alone.