18

So I'm a few months into my PhD. It's built on top of previous work that has not been published. My supervisors, collaborators, and myself are working on to get this published. Now here's where things get a bit weird for me.

I found some problem with the previous work. Discussed the problem with my supervisors and have fixed them. For the publication, I'll need to redo the experiment.

So far I'm not getting a satisfactory result as it's tricky to get right. Not gonna go into why, it's just tricky. My collaborator has been impatient with the result I'm getting. So now he has decided to rerun the experiment himself and double checking that he's getting the result I'm getting. Turns out it's quite similar. So now he's trying his own idea he has only briefly disclosed to me and one of my supervisor. When asked indirectly why he's doing this, he said he's just trying to understand the experiment better.

I found this disturbing. Why are you repeating my experiment yourself only to come and tell me you found exactly the same problem and result?

What's everyone take on this? I would like others opinions first before I discuss the issue with my supervisors.

Improve this question
13
  • 28
    I don't see a problem with this. Maybe your supervisors thought the idea will pan out, and just want to make sure you have not missed anything. Aren't three heads better than one?
    – Prof. Santa Claus
    Nov 9, 2017 at 3:43
  • 71
    The entire point of scientific testing and publication is to increase the knowledge people have, and to let others test your work for reproducibility. The collaborator is doing both - being able to say "independent reproduction of the experiment obtained similar results, so the authors changed ... and hence were able to ..." in the paper is a useful addition. It would only be suspicious if they did not tell you about the retest and alterations.
    – Nij
    Nov 9, 2017 at 6:43
  • 54
    Like others here, I'm perplexed (and a little disheartened) by the fact that a PhD candidate would view independent replication of their research as "disturbing". Why do you see it that way?
    – Nuclear Hoagie
    Nov 9, 2017 at 13:46
  • 19
    While Nij and Nuclear Wang essentially wrote the most important points here, I would like to add that I, personally, would be very satisfied if someone rerun my experiment and got exactly the same result.
    – Maciej
    Nov 9, 2017 at 14:31
  • 7
    I think one thing isn't clear. Is your collaborator getting the same UNSATISFACTORY results that you are getting? Independent reproduction of unsatisfactory results can just as meaningful as reproduction of satisfactory results. I would be concerned that what you consider satisfactory may not be reproducible because it is the result of random errors, and not because "it's tricky".
    – Jim
    Nov 9, 2017 at 20:49

8 Answers 8

Reset to default
120

I don't find that behavior strange at all. If I want to understand problems with algorithms (I don't do experiments with anything other than algorithms) I need to run them by myself and sometimes even implement the method from scratch. It's not that I do not trust other people's code, but if I want to understand all the mechanisms well, it's much easier to get a grip on it when I've done it myself.

The other way round, I do not have any objections if a coauthor of mine re-implements a method I already provided just to see how it works. On the contrary: Usually I benefit from this in one way or another, e.g. by learning new tricks or getting rid of bad programming style…

Improve this answer
1
  • 12
    I can not upvote this enough as rewriting algorithms either by myself or co-authors is such an important aspect of producing an 'as good as possible' algorithm for the final manuscript/paper/study.
    – Bas Jansen
    Nov 9, 2017 at 15:21
72

When I was a graduate student, I was working together with another student who implemented a rather complicated numerical simulation. To understand it better, and to do some additional experiments with this model, I decided to re-implement it myself. I was not able to get the same results as the other student. After a thorough review of my code, I found a minor mistake, and corrected it, but I still wasn't able to reproduce the other student's results.

In the end it turned out that the other student also had multiple mistakes. We fixed them, and finally everything matched up.

We had two independent implementations of the model we were studying, and finally they were giving the same result. Only at this point was I fully confident that our results were correct.

Double-checking makes a lot of sense. Mistakes happen, quite frequently. I would never trust a single result, either mine or someone else's, nearly as much as an independently verified experiment.

I have also worked in an environment where it was frowned upon to double check a coworker's work, claiming that "it is a waste of time". In reality, people were afraid that their work would be overtaken or stolen, and they were withholding data in order to prevent others from checking their result. In the end, it turned out that some results I was relying on were plainly wrong, which resulted in even more waste of time in the end. My advice is to try to avoid creating a toxic environment full of jealousy, like the one I experienced. Double checking is always a good thing.

Improve this answer
2
  • 1
    In the last years there have been quite a few papers where experiments couldn't be reproduced later. Some people wouldn't want you to try to reproduce their experiments because they know they can't be reproduced.
    – gnasher729
    Nov 12, 2017 at 16:17
  • 1
    Great answer, and this is exactly why we shall be a bit disturbed by papers using Wolfram software, given that they claim something like "implementation is not important, you just want to know the result".
    – Andrea Lazzarotto
    Nov 13, 2017 at 1:04
17

I don't see any problem with your collaborators wanting to be sure of things. I also don't work in the experimental sciences but any time one of my collaborators emails me a page of mathematics, the first thing I do is check it line by line to make sure it's really true, and the second thing I do is try to improve it.

It sounds like there's a bit of a communication problem between you and your collaborator. Also, if your collaborator is an established researcher, they might be being a little impatient with you and forgetting that, as a brand new PhD student, it's going to take you a little longer to get things sorted out.

Talk to your advisor, as always. If you're upset by what your collaborator has done, tell your advisor about it. They'll either explain that things are OK, or agree with you and talk to your collaborator about it.

Improve this answer
12

Is it normal for collaborator to rerun experiments himself?

Not only is it normal, I'd say it is highly recommended if you are able to do so. If not, either be extremely careful about verifying the vailidity of your coauthor's experimental procedure or qualify the experimental results' validity.

I'd also recommend that you endeavour to make your own experiments easily reproducible, so that your coauthors (and paper audience) can repeat them.

Improve this answer
6

I am glad that your collaborator is coming up with the same results.

Are you concerned that your collaborator is attempting to take your ideas and add his own ideas in order to create a new derivative work? In the setting you describe this sounds like it might be difficult for him to accomplish.

On the other hand, many of the people that I have worked with are too lazy to run experiments independently. I might be a little bit concerned in your position.

My advice is to make sure that everyone you are working with are aware of his work, try to keep everything in the open so that it is harder to work against you, and to get published as quickly as you can, without sacrificing quality of course.

Improve this answer
1
  • You took the words out of my mouth. He did not make his intention clear and was not sharing results with us unless i ask him to. Hence I was suspicious why would he do that.
    – woof
    Nov 15, 2017 at 0:49
5

You want as many people as possible tackling this problem: you, your collaborator, your advisor. If there's a potential problem with the result/experiment/paper you want to know all about it and fix it before you publish and have folks either correct you or rely on something that turns out not quite right years later.

You want those people working together (to help each other) and separately (so group-think doesn't shut out ideas(.

Improve this answer
4

Well, scientific experiments are always subject to being repeated and cross-checked by others, so that in itself is not a problem. That's how science works. As for the etiquette of a "collaborator" re-running and double-checking your results, I guess that that all depends on the details. You didn't give much information on this collaborator. Who is he or she and does he or she have a legitimate and valid interest in seeing the experiment properly completed in a timely manner? If, for example, he or she is to be a co-author on a paper based on the experiment, perhaps that person has a legitimate interest in seeing the experimental problem resolved quickly. If that's the case, then I don't see any black-or-white answer to the problem. May be something that you, the person, and a supervisor will all have to iron out. Finally, if the person is pitching in to investigate the experimental problem, it may be a bit damaging to your ego but you should also realize that that person may be able to help in identifying the problem so that you can go on and graduate.

Improve this answer
3

I would consider it unusual in many circumstances, you generally should trust your collaborators enough that you don't need to redo their experiments. It can be very useful to perform independent experiments to get higher confidence in the results, but that would be something that you discuss prior to doing the experiments.

In your case there is an important aspect that changes this. Your experiments aren't working correctly yet. Curiosity about this particular problem with the experiments is a plausible explanation for trying to reproduce your results. And from what I read, they're trying to find a way to fix this issue, and reproducing your results would be the logical first step for that.

You should consider this as useful information, as an independent reproduction also excludes a lot of sources for errors as your equipment, material and people are likely different. So this could help you to find the issue with your experiment quicker.

There is a communication issue here, and as already said in another answer, you should involve your supervisor to resolve that.

Improve this answer
3
  • Agreed. Actually, it would be useful to know if the collaborator is in a different university. If they were in the same university, it would probably be better to get the experiment working by going to the student's lab and working together. But, if they're in a different city, that doesn't work so well and they could only really do it on their own.
    – David Richerby
    Nov 9, 2017 at 11:17
  • 4
    Actually it depends a lot on the domain. If it is computer science, and experimentation means to run some software, it is very common (and should be expected) to redo the experimentation. If it was biology, and experimentation requires a complex & costly apparatus, things are different (but I don't know how)
    – Basile Starynkevitch
    Nov 9, 2017 at 12:13
  • 3
    @BasileStarynkevitch My point of view is from a biological/chemical domain, and the way the question is written does read more like experiments in a lab than running code on a computer to me, but it is not actually stated unambiguously in which domain the asker is working.
    – Mad Scientist
    Nov 9, 2017 at 14:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .