19

I found a useful advice if I cannot reproduce computational published results in (What should you do if you cannot reproduce published results?). However I would like to ask the same situation with experimental results.

I am from chemistry, and it often involves synthesis of compounds. I know every lab in the world cannot be identical even if all equipment are the same, because many external factors such as climate, humidity, lab temperature, etc. may possibly interfere with the experiment.

If I have followed all the steps described in the literature, from the source and purity of chemicals to every single step described, but still I cannot reproduce the results (evidence from some characterization techniques), what can I do? Should I contact the author to give me all 'hidden' steps like the size of beaker, etc.? Or the journal reviewers think that the authors have already disclosed enough information for others to reproduce it, so the remaining job to investigate how to reproduce the results is on me? However sometimes the description is really vague, like 'drying in a vacuum oven at 50oC for 10 minutes', but what is the degree of vacuum?

(Sometimes the size of beaker may play a role, as evaporation way affects its crystal size, also the position of drying compounds in oven, etc.)

2
  • 4
    I think the advice from the linked question applies even here. Try to replicate the experimental conditions as much as possible, try to get any "hidden" steps if there are any, and if the results are still a long way apart, then you have two options: a) try to publish your findings in a rebuttal/correspondence, or b) send a letter to the editor explaining that you have concerns that the explained experiment and the reported results do not match.
    – posdef
    May 11, 2014 at 10:32
  • 1
    Also keep in mind that published results, even widely cited ones, are sometimes just wrong. Just ask Thomas Herndon who found multiple flaws in an economics paper that was taken to be canon. So, try your best to replicate the experiment but if you are not getting the expected results, do not be overly surprised.
    – wsaleem
    May 11, 2014 at 20:05

3 Answers 3

20

@posdef's comment is already very good.

I would definitely suggest contacting the authors and trying to get exact agreement also on the details that they didn't describe in the original paper. The original authors should be quite interested in an independent replication, so they should have an incentive to collaborate. Depending on how much they contribute, they could also hope for co-authorship on your paper.

Conversely, if they are not cooperative, that is another piece of information that should be included tactfully (!) in the paper you are putting together.

If you are lucky in your analyses, you may be able to identify the exact circumstance or unreported detail that makes the difference between reproducibility and non-reproducibility, and there you have the nucleus of an original paper all by itself.

On the other hand, if you cannot reproduce the results even with help from the original authors, this should be worth at least a correspondence. Reproductions are important, and there are far too few of them!

I part ways with @posdef's comment in that I would definitely not send a letter to the editor voicing concerns about reports and reproductions not matching. This already points towards a suspicion of scientific wrongdoing on the part of the original authors. And unless you have a lot more to go by, a simple inability to reproduce a finding should not be taken as the result of foul play. There are so many potential reasons for non-reproducibility, from the ones you mention to simple random error (aka, uncontrolled conditions), that there is really no need to even imply dishonesty. Such an implication can mess up either the original authors' or your own reputation in the field for decades, so it should not be made lightly.

1
  • Just to make it perfectly clear, my option (b) that is writing to the editor, is merely an alternative. I never claimed that it's the best one, or the right one :)
    – posdef
    May 11, 2014 at 18:03
11

There is nothing wrong or rude about contacting the author and asking for some help in this matter.

You should identify not only who you are, but what your institutional affiliation is (and as I guess you are a student or postdoc who you are working with/for).

You should explain what you are up to (simple replication, or are you trying to use this work as a step in a larger process), an outline of what you have tried, what choices you made on any steps that were not completely specified, and how your outcome has been unsatisfactory (it is possible that the author needed multiple tries to get this to work and has seen the same failure mode before).

-2

Easiest solution is to invite them over and let them do the experiment. You will find what they are doing different and then you can decide together how to proceed with publishing the results.

4
  • 7
    Easiest solution is to invite them over and let them do the experiment. This hardly sounds like the easiest thing to do.
    – Mad Jack
    May 11, 2014 at 22:02
  • depends where they live. But trying to get these details right by email or phone can be very difficult in experiments involving real world artefacts/substances/hardware (as opposed to just computer code for example). It may be some noise or uncompensated variable in your environment or set up, or vice-versa, or both. So if having them over doesn't solve the problem, hopefully they will invite you back for a reciprocal visit to recheck with their apparatus and environment. May 12, 2014 at 11:55
  • 1
    "Just computer code" can be surprisingly hard to reproduce. Some colleagues once tried to reproduce a published paper, with the full support of the original authors over many iterations, with a lot of back and forth, but they never managed an exact reproduction. There were just too many undocumented issues with data cleansing and so forth. Search for the abstract by Syntetos, Boylan, Mohammadipour & Goodwin here: forecasters.org/wp/wp-content/uploads/… May 13, 2014 at 19:10
  • @StephanKolassa and don't get me started with compiler and library versions...
    – Davidmh
    Aug 25, 2015 at 9:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .