60

For the last few days, I have been trying to reproduce things in a published paper (in quite a reputed journal) from my field. My postdoc mentor is the last author of the paper and the main work was carried out by her collaborators from a different institute and a first author is finishing his Ph.D.

When I saw the paper, I immediately got skeptical because they had missed some obvious things, so I rewrote all the code from scratch and, after a week's extensive checking and cross-checking, confirmed that their results are completely wrong. I told two of my trusted colleagues and they asked me to tell my mentor this. However, I am quite uncomfortable and confused about this.

The student who is the first author has based a big part of his later work on these results. This means that the later work by the student might also be completely wrong. My question is how to proceed from here? Can I ask my mentor to retract their paper? Or should I ask her permission to write a comment about the paper? If I do either of these (or something else which reveals that those results are wrong), would I be considered guilty of destroying the thesis of the first author guy?

I really appreciate any type of help in this matter. Thanks in advance.

Edit: All the answers and comments here are very helpful and I thank you all for them. However, I think I was slightly vague while asking. I very well understand that I could be certainly wrong. But the question is what do I do if I am not? In that case, what should be done? (As things stand, they have surely got at least one plot wrong because it directly goes against a famous fundamental theorem. I got this confirmed from a few people in the field.)

  • 72
    Why so drastic? Why not just talk with your mentor about your concerns? Remember that you might be wrong too. – Bitwise Sep 11 '16 at 12:35
  • 43
    Majid's answer is absolutely correct. The key thing here is not to go in with statements like "missed obvious things" or "results are completely wrong." Sit down with her and ask for help reconciling your results with the paper's. Several people (from the sound of it further into their education and career than yourself) worked on the paper. It was peer-reviewed before publication. While it's entirely possible you might be correct, it's (with respect) more likely that you've misunderstood something. – T.J. Crowder Sep 11 '16 at 15:55
  • 18
    "I very well understand that I could be certainly wrong. But the question is what do I do if I am not?" -- that's completely the wrong way to think about it. You don't know whether you're right or wrong, but you must act anyway without certainty. Therefore, you do not need and could not use a course of action that's applicable only if you're right. To the best of your knowledge and ability you're right. Next step is for you and your mentor to figure out whether, to the best of your combined knowledge and ability, you're right. – Steve Jessop Sep 11 '16 at 18:52
  • 12
    That is, you need to know how to approach your mentor with a possible problem in their result. You don't need to know how to approach your mentor with a certain problem in their result. – Steve Jessop Sep 11 '16 at 18:57
  • 10
    Viz your edit, I think it does not matter how certain you are that you are correct at this point. I think your actions should be the same in any case. Just approach the mentor and ask her to clear up your doubts, while avoiding framing them as absolute (that is to benefit both your ego -- if you are wrong -- and hers -- if you are right -- as well as your relationship). If she tries to deny your doubts without clearing them up -- well, then your certainty about the matter would be very important in deciding what to do next, but that is a different question altogether. – tomasz Sep 11 '16 at 20:31
108

As a matter of fact, I believe there is no such thing as guilt or ruin for other people's work when it comes to the academic aspect of the knowledge. Actually, if you do not share this with your mentor in order to be fixed, you may ruin many other further research...

I suggest you do this respectfully and carefully. You can go to your mentor and talk about this problem. You can show her your code and your understanding from the paper, then confirm whether or not you are right. Let her decide what she can do about this. There is also a slight chance you have made a technical mistake while investigating their work, so keep this in mind when you discuss the issue.

  • 10
    Yes the answer is quite satisfactory, you can approach her and ask for their code and tell her that i am trying to reproduce the results but there are some differences which i can not sort out, and it will br nice of you if you can help me in. In that case she wont feel bad and if there is something wrong, that can be corrected as well – Shahensha Khan Sep 11 '16 at 16:27
  • 27
    I would add that it's really useful, when someone approaches you to say that you're wrong, if they can provide some insight into what went wrong. If they say "I wrote the code from scratch and got a different result" then you know you can't both be right, but it doesn't tell you anything about which of you (or both) is wrong. On the other hand, if they say "your paper states, and therefore I suppose your code probably relies on, [this obviously wrong thing] and here's a paper that proves the opposite of what you assumed", then you can immediately relate what they're saying to your paper. – Steve Jessop Sep 11 '16 at 19:01
  • 13
    (I speak not as an academic, but as someone who writes code for a living. I have written a lot of wrong code that has needed correction either by myself or others). – Steve Jessop Sep 11 '16 at 19:03
  • 2
    I would say that it's possible to ruin other people's work. I'd also say that it is necessary to ruin wrong work, for the reason you said, it might ruin other future research otherwise. – Turion Sep 12 '16 at 10:48
  • 3
    It's the duty of people who work in science to try to reproduce the published work of others once in a while: that's why people publish their work. It's also their duty to reveal their success or failure when they do try to reproduce published work. (Science --the pursuit of knowledge-- suffers when irreproducible work goes unchallenged.) So this answer is correct. – O. Jones Sep 14 '16 at 16:17
51

I suggest you treat this as a learning process. After all, you have spent on this only a week, while the authors have spent significantly more, so there is a chance you are wrong.

So, with this working theory in mind, present your results to your mentor and discuss with her what you think is wrong with their work and why and ask her to explain what you could be missing.

After the discussion you should, ideally, either realise why you were wrong; convince them of what they did wrong; or all of you agree to disagree on the correctness of the results. In the second and third scenario, your mentor should be able to suggest some path.

I do not think you should be asking them to retract the paper unless they have shown gross negligence in the research process or outright cheated; it should be up to them.

  • 13
    I agree completely. The OP's job is to understand why his recent computations produce results that are at variance with his mentor's published work. He should talk to the mentor focusing on that. Things will go naturally from there. If the explanation is that the OP's computations are correct and the work of the mentor is not, then the mentor will almost certainly not need the profound implications of that pointed out to her. Just seek the truth, one step at a time. – Pete L. Clark Sep 12 '16 at 1:51
  • 1
    Agree to disagree is good in political disputes, not when discussing things that can be experimentally verified. – Tomáš Zato Sep 13 '16 at 11:28
  • 1
    @TomášZato unfortunately, that is not always the case, because the experiments would be prohibitive to run in a short timescale, or because the disagreement is in the interpretation. – Davidmh Sep 13 '16 at 12:26
41

I think there is a simple answer. From what you wrote it sounds like you rather "could not reproduce the results" than "proved that the results are wrong". While the first is often an indication for the second, this need not be the case (especially since you "rewrote the entire code").

Now focus on a solution instead of the problem. What you should do is: Approach your mentor with "I tried to reproduce the findings of the paper in this and that way and ended up with different conclusions." From there on the story may evolve differently: It may be that there is a misunderstanding about the "missed obvious things", it may still be bugs in your code, it may that the paper was ambiguous about some things, also it may be that there is something wrong...

In this discussion be sure to be open minded and free of prejudice. As an example: When somebody tells me something in a scientific discussion that I think is wrong or bogus, I get a much better and helpful response when I say "Sorry, I did not understand that, can you say it differently?" than "That's nonsense."

  • 4
    This - even if you believe (or even know) you are right, it is often easier to get someone to engage by asking for help with where you went wrong. Let that person realise they are wrong rather than start by telling them they are wrong. – JenB Sep 12 '16 at 9:32
  • 2
    +1: The "I tried to reproduce your results and couldn't. Where did I go wrong?" approach is perfect. It doesn't go in with a "you're wrong" attitude, it asks for help, and it allows the mentor to say, "Hey, looks like there is a problem in the original", IF they decide that this is the case. – Wayne Sep 13 '16 at 17:35
19

I would recommend that instead of talking directly about the flaws of their work, you take an approach of explaining what you did.

Here's an example:

Hello Ms Mentoress, I have a question about that paper you published with co-authors A, B, and C a while ago. So I was trying to reproduce the results with code I've written on my own and I couldn't get it to work. Can you have a look and help me with this?

There are several reasons for using this approach:

  1. You are not blaming anyone for anything, and you are not being aggressive. Retractions and the likes are not under discussion now, and they shouldn't be until the mistake is understood, by all parties involved.

  2. There is a small possibility that you are wrong and not them. This will save you the embarrassment of blaming someone for being wrong while in fact they were right all along.

  3. In the case that you are correct, your mentor will realise on her own that she was in fact wrong. She will hold you in high esteem for this, because in her mind you are now a scientist who can rigorously test other people's work, which in this case happened to be hers. This will build bridges instead of burning them

  4. In the case this results in a comment or discussion paper, there's a good chance you will be co-author with that group of people who will most likely try to correct themselves. As these are probably people with some reputation who have published in high ranking journals, this will put you together with people you want to be in good relations with, not against them.

  • Is there the chance it has to do with deprecated hardware no longer available? the OP didn't mentioned what the code was about, so it is likely it could be some kind of benchmark and we know benchmark change over time – GameDeveloper Sep 12 '16 at 9:44
  • 3
    @DarioOO It seems strange for the OP to try to replicate a benchmark on different hardware than that described in the paper and then become confused when the result is different. – Milo P Sep 12 '16 at 16:34
4

From personal experience, I know that it may be very difficult to replicate results. I would recommend first asking yourself the following questions: Can I explain in a sentence or two what drives your mentor's results? Can I do the correction that gets results I believe to be correct? Are these results central to the paper?

Allow me to elaborate: You need to be able to say "You get result Y only because you did X. But X is incorrect methodology because Z. When I try to replicate your result, instead of X, I do X', and then your result Y goes away. This is important because your result Y is central to the paper."

If you can fill in the variables in these sentences, then it may be time to talk to the mentor. It is then a matter of being diplomatic. Make sure that they understand that your intention to discuss these with your mentor only, and absolutely no third party. (Don't say this directly, but let this be your approach and let that show.) I would also recommend de-personalizing the issue in grammar: Say "these results" rather than "your results".

Finally, should you even talk about this? The decision is up to you and there is no clear answer; but it is a function of the answers to the following questions: How many years have you known your mentor? How good are you at being a diplomat? How insecure is your mentor as a person? What do you have to gain or lose in case of go / no go?

  • 2
    I would take out the last paragraph. Other than that -- strong, helpful answer. – aparente001 Sep 14 '16 at 0:46
3

Right now, you don't know that your mentor's paper is "wrong." What you do know is that your results differ from hers.

Because she is your mentor, I would approach her in this spirit, and say, "I was trying to replicate your paper, and came up with entirely different results. One of us is doing something wrong, and it's probably me. Could you explain why your results are so different from mine?

It's possible that she'll point out some error in your logic or code, and you'll be saved the embarrassment of making a false accusation. It's also possible that you are correct, and by treading softly, you will keep her on your side. If you are right, you will give her the chance be a "hero" by contacting the others involved with the paper and suggesting that it needs to be corrected. Inwardly, she will be grateful to you for saving her from embarrassment.

3

Can I ask my mentor to retract their paper?

You can ask her anything. But you shouldn't. Don't tell her what to do; rather, tell her what you have found. If the paper is wrong, you should assume that the authors will behave with integrity and take the appropriate action: they almost always will.

A retraction is absolutely the last step, which will only happen if the paper really is wrong and it can't be corrected. It's not something you discuss the first time you mention that there seems to be something wrong: that would be a bit like beginning a conversation with, "I can't find my phone. Have you seen it? The maximum penalty for theft is ten years in jail."

Or should I ask her permission to write a comment about the paper?

No. On a technical level, you don't need anybody's permission to write and publish about their work. However, if you do end up publishing something saying "X is wrong", it's polite to let the authors of that work know in advance. It can also save a lot of time, because it means that there are more people working on the fix (you and them) and some of them are the world's greatest experts on that particular thing (them: even if they're the world's greatest experts on a wrong thing, they know much more about what they did than you do).

But you're not ready to write anything, yet. Talk to your mentor about what you've found and try to come up with a solution together. The only reason to continue without her is if she refuses to cooperate. In that case, because you're at an early stage of your career, I'd strongly recommend that you find somebody more senior to work with, since writing "X is wrong" papers is rather delicate.

If I do either of these (or something else which reveals that those results are wrong), would I be considered guilty of destroying the thesis of the first author guy?

No, not at all. Just as you're not guilty of destroying somebody's freedom if you call the police when you see them committing a crime. If people produce work that is incorrect, that is their fault alone.

  • @LeonMeier I've written a paper that says that X is wrong. But, in mathematics, one can at least make such a statement objectively. – David Richerby Sep 18 '16 at 19:32
  • The paper has been published. X was a published claim made by a group of people, which was backed up by an erroneous proof. – David Richerby Sep 18 '16 at 19:35
  • Of course it was serious/reputable and peer-reviewed. The suggestion that I might publish somewhere non-serious/non-reputable/non-peer-reviewed is, er, not very complimentary. – David Richerby Sep 18 '16 at 21:38
  • @LeonMeier Apologies for being unclear but I was somewhat insulted by the suggestion that I'd publish my work in any other way and refer to it as a "published paper". – David Richerby Sep 19 '16 at 8:45
  • I wouldn't describe a tech report or somebody else's work as a published paper. – David Richerby Sep 19 '16 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.