Since reviewers don't check the experimental results by trying to reproduce the experiment, is it possible for someone to submit a paper which basically says "Method X was proposed in paper Y and according to them it improved performance by 15% as compared to baseline. However when we tried it, it didn't work so well (only 2% improvement). Hence we propose its modification which actually achieves 14% improvement as compared to baseline on the same train/test data."?
-
3All the time. And this should be obvious. I mean, even if you use an appropriate method and do everything right, if you publish circa 2 sigma (or approx. 95% confidence) results then you will simply be wrong one time in twenty through no fault of your own. Nineteen times out of twenty times anyone who tries to reproduce one of these results will get a different answer. This is elementary statistics.– dmckee --- ex-moderator kittenMar 20, 2013 at 15:02
-
1I posted an answer to a similar question a while back that may be worth looking at.– eykanalMar 20, 2013 at 15:19
-
1@dmckee: It's obvious that it should happen at least 5% of the time, but my impression is that it happens a LOT more often than that.– JeffEMar 20, 2013 at 15:27
-
1@user13107 You will be hard pressed to find a published paper that does nothing more than (fail to) replicate a previous study; most papers examine something similar and compare results. As for examples, I recently posted this answer to Skeptics.SE highlighting different findings between research studies.– eykanalMar 20, 2013 at 15:55
-
1@JeffE: pls see my answer, while I doubt the 5%, there are reports confirming (by systematically trying to reproduce literature findings) your impression - in fact, the two comments I linked could not reproduce 100 out of 120 papers (i.e. false discovery rate 75 - 90 % [95% c.i.]).– cbeleites unhappy with SXMar 20, 2013 at 21:57
5 Answers
- Make sure the difference comes from the experimentation not from the adopted technique/method.
- Make sure you have the same settings as the other paper. Sometimes people make assumptions for the sake of simplicity in experimentations. For example, I remember I did experimentation assuming acyclic graph exists.
- Do you have some kind of randomness (i.e. generating random instances of the problem)? If yes, revise its output. Sometimes you examine easy instances while others base their experimentations on hard instances of the problem.
- In some areas, there are benchmarks and robust solvers for particular problems/structures. If your field have benchmarks, try to compare your method against it.
Either way, I am sure you have important parameters to control the experimentation (i.e. number of variables..etc). check their role.
Most importantly, you need to theoretically justify why your method will save 14% while other method saves only 2% in practice.
-
3These are worthy goals, but are not always possible in practice. Unfortunately, the prior paper may not have exposed all the details of its implemented method, nor the precise settings of its experiments, nor the source of its input data. Don't make the same mistakes. The last point is especially difficult; there may be no theoretical justification for why method X works better than method Y in practice, because there is no crisp, accurate, abstract formulation of the phrase "in practice".– JeffEMar 20, 2013 at 15:30
-
@JeffE I totally agree with your comment; that's why I see many evaluating papers in the literature of some fields trying to find which method better than another in practice. Mar 20, 2013 at 15:46
Yes. To improve on others findings is a common situation. The fact that the first paper overstated performance may not necessarily be wrong from th epoint of their experimental setup but they may have missed some component that negatively affected their experiment. I would say that this reflects incremental improveents
in the development of ideas in science. As someone once said: "If I knew what I was doing, it wouldn't be science".
Some fields deal with exact numbers in which case you don't have a contradiction, you have identified an error. When you are dealing with inexact numbers that have "measurement error", you need to be careful. As much as I dislike statistics, they can be, and really are, your friend when dealing with measurement error.
You say Paper Y found that Method X was 15% better than baseline. Did they do a statistical comparison to see if Method X was better than baseline, or did they calculate confidence intervals and really say that it was 15%+/-0.000001 better than baseline? Is your 2% difference from baseline statistically reliable? Is your 2% difference from baseline statistically different from 15%? Then we have your statements about the modified methods. Is the 14% statistically reliably different from the 2% improvement you saw?
If there is measurement error then all you can say is that it is extremely unlikely that your implementation of their method is the same. This doesn't really contradict them, and it definitely doesn't say they are wrong.
-
1it definitely doesn't say they are wrong — True, but it also definitely erodes any confidence that they are right.– JeffEMar 20, 2013 at 15:33
JeffE commented:
It's obvious that it should happen at least 5% of the time, but my impression is that it happens a LOT more often than that.
The 5% aren't all that obvious to me: if I understood correctly, the 5% are the (in)famous p-value.
That is, of every 100 false null-hypothesis, 5 are rejected ("we found something") by mechanically rejecting H0 when the p-value indicates that the probability of observing such or more extreme results as we got reaches 5%.
| what the paper does | | reject H0 not reject H0 | sum ------------------------+---------------------------+------ truth | null hypotesis | 5 95 | 100 v alternative h. | ? ? | ? ------------------------+---------------------------+------ sum | ? ? |
The number of contradicted papers, however, should depend on the number of falsely accepted hypotheses among all accepted hypotheses (whether true or not). The problem is, we'd need to know the number of correctly accepted alternative hypotheses to calculate which percentage should lead to contradictions.
This we don't know, but of course it depends on the number of true alternative hypotheses among all hypotheses, which we may call the "prevalence of good ideas".
If we stay in analogy to medical terms, the percentage of contradicted papers should be (1 - predictive value of rejected null-hypotheses). And this will be much larger than 5% if lots of "bad" ideas are tested.
Literature:
- Ioannidis, J. P. A.: Why most published research findings are false. PLoS medicine, 2005, 2, e124
- Der Schein der Weisen [popular science; in German]
Here are two comments from pharmaceutical companies reporting on the issue for (mostly oncological) drug development:
Prinz, F. and Schlange, T. and Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov, 2011, 10, 712
Only for about 1/5th (14/67) of the projects, the reports from literature could be reproduced, of which 1 was reproduced directly, 12 after some kind of adaption.Begley, C. G. and Ellis, L. M. Drug development: Raise standards for preclinical cancer research. Nature, 2012, 483, 531-533
Confirmed findings from 6 out of 53 "landmark" papers. The authors also report how often the studies got cited: no difference between the studies they could not confirm and the confirmed ones (huge spread, if there's a difference, non-reproduced articles got cited more often).
-
2p-value, i.e. the probability to falsely reject a true null-hypothesis - This is not the p-value! Instead it is one of its numerous misinterpretations.– BravoMar 20, 2013 at 22:14
-
@Shyam: please edit to correct me! But note that I'm talking about a (large) number of studies where the null-hypothesis is actually true (100 in my example). (And you should have told me that there was a wrong "accept" just below.) - so please proofread again ;-) Mar 20, 2013 at 23:55
-
1
It is completely commonplace, particularly in finance publications, to fabricate results.
A previous thesis adviser of mine actively encouraged not reporting results which did not support the story he was trying to tell, and to completely change test design and the statistical tests performed when it was possible to get results which did support the story.
It should come as no surprise that papers report results which contradict each other.
-
-
3This is a fairly serious charge to bring against an entire field. Do you have anything (non-anecdotal) to support this claim?– eykanalMar 21, 2013 at 12:33
-
Not bringing a claim against an entire field, just mentioning that it is common. Not sure how anyone could deny that...– H. D.Mar 21, 2013 at 20:29
-
5Sorry, but "just mentioning" that obviously unethical behavior is "completely commonplace" IS bringing a serious charge against an entire field.– JeffEMar 22, 2013 at 1:43
-