Since reviewers don't check the experimental results by trying to reproduce the experiment, is it possible for someone to submit a paper which basically says "Method X was proposed in paper Y and according to them it improved performance by 15% as compared to baseline. However when we tried it, it didn't work so well (only 2% improvement). Hence we propose its modification which actually achieves 14% improvement as compared to baseline on the same train/test data."?
- Make sure the difference comes from the experimentation not from the adopted technique/method.
- Make sure you have the same settings as the other paper. Sometimes people make assumptions for the sake of simplicity in experimentations. For example, I remember I did experimentation assuming acyclic graph exists.
- Do you have some kind of randomness (i.e. generating random instances of the problem)? If yes, revise its output. Sometimes you examine easy instances while others base their experimentations on hard instances of the problem.
- In some areas, there are benchmarks and robust solvers for particular problems/structures. If your field have benchmarks, try to compare your method against it.
Either way, I am sure you have important parameters to control the experimentation (i.e. number of variables..etc). check their role.
Most importantly, you need to theoretically justify why your method will save 14% while other method saves only 2% in practice.
Yes. To improve on others findings is a common situation. The fact that the first paper overstated performance may not necessarily be wrong from th epoint of their experimental setup but they may have missed some component that negatively affected their experiment. I would say that this reflects
incremental improveents in the development of ideas in science. As someone once said: "If I knew what I was doing, it wouldn't be science".
Some fields deal with exact numbers in which case you don't have a contradiction, you have identified an error. When you are dealing with inexact numbers that have "measurement error", you need to be careful. As much as I dislike statistics, they can be, and really are, your friend when dealing with measurement error.
You say Paper Y found that Method X was 15% better than baseline. Did they do a statistical comparison to see if Method X was better than baseline, or did they calculate confidence intervals and really say that it was 15%+/-0.000001 better than baseline? Is your 2% difference from baseline statistically reliable? Is your 2% difference from baseline statistically different from 15%? Then we have your statements about the modified methods. Is the 14% statistically reliably different from the 2% improvement you saw?
If there is measurement error then all you can say is that it is extremely unlikely that your implementation of their method is the same. This doesn't really contradict them, and it definitely doesn't say they are wrong.
It's obvious that it should happen at least 5% of the time, but my impression is that it happens a LOT more often than that.
The 5% aren't all that obvious to me: if I understood correctly, the 5% are the (in)famous p-value.
That is, of every 100 false null-hypothesis, 5 are rejected ("we found something") by mechanically rejecting H0 when the p-value indicates that the probability of observing such or more extreme results as we got reaches 5%.
| what the paper does | | reject H0 not reject H0 | sum ------------------------+---------------------------+------ truth | null hypotesis | 5 95 | 100 v alternative h. | ? ? | ? ------------------------+---------------------------+------ sum | ? ? |
The number of contradicted papers, however, should depend on the number of falsely accepted hypotheses among all accepted hypotheses (whether true or not). The problem is, we'd need to know the number of correctly accepted alternative hypotheses to calculate which percentage should lead to contradictions.
This we don't know, but of course it depends on the number of true alternative hypotheses among all hypotheses, which we may call the "prevalence of good ideas".
If we stay in analogy to medical terms, the percentage of contradicted papers should be (1 - predictive value of rejected null-hypotheses). And this will be much larger than 5% if lots of "bad" ideas are tested.
- Ioannidis, J. P. A.: Why most published research findings are false. PLoS medicine, 2005, 2, e124
- Der Schein der Weisen [popular science; in German]
Here are two comments from pharmaceutical companies reporting on the issue for (mostly oncological) drug development:
Prinz, F. and Schlange, T. and Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat Rev Drug Discov, 2011, 10, 712
Only for about 1/5th (14/67) of the projects, the reports from literature could be reproduced, of which 1 was reproduced directly, 12 after some kind of adaption.
Begley, C. G. and Ellis, L. M. Drug development: Raise standards for preclinical cancer research. Nature, 2012, 483, 531-533
Confirmed findings from 6 out of 53 "landmark" papers. The authors also report how often the studies got cited: no difference between the studies they could not confirm and the confirmed ones (huge spread, if there's a difference, non-reproduced articles got cited more often).
It is completely commonplace, particularly in finance publications, to fabricate results.
A previous thesis adviser of mine actively encouraged not reporting results which did not support the story he was trying to tell, and to completely change test design and the statistical tests performed when it was possible to get results which did support the story.
It should come as no surprise that papers report results which contradict each other.