14

Since reviewers don't check the experimental results by trying to reproduce the experiment, is it possible for someone to submit a paper which basically says "Method X was proposed in paper Y and according to them it improved performance by 15% as compared to baseline. However when we tried it, it didn't work so well (only 2% improvement). Hence we propose its modification which actually achieves 14% improvement as compared to baseline on the same train/test data."?

  • 3
    All the time. And this should be obvious. I mean, even if you use an appropriate method and do everything right, if you publish circa 2 sigma (or approx. 95% confidence) results then you will simply be wrong one time in twenty through no fault of your own. Nineteen times out of twenty times anyone who tries to reproduce one of these results will get a different answer. This is elementary statistics. – dmckee Mar 20 '13 at 15:02
  • 1
    I posted an answer to a similar question a while back that may be worth looking at. – eykanal Mar 20 '13 at 15:19
  • 1
    @dmckee: It's obvious that it should happen at least 5% of the time, but my impression is that it happens a LOT more often than that. – JeffE Mar 20 '13 at 15:27
  • 1
    @user13107 You will be hard pressed to find a published paper that does nothing more than (fail to) replicate a previous study; most papers examine something similar and compare results. As for examples, I recently posted this answer to Skeptics.SE highlighting different findings between research studies. – eykanal Mar 20 '13 at 15:55
  • 1
    @JeffE: pls see my answer, while I doubt the 5%, there are reports confirming (by systematically trying to reproduce literature findings) your impression - in fact, the two comments I linked could not reproduce 100 out of 120 papers (i.e. false discovery rate 75 - 90 % [95% c.i.]). – cbeleites supports Monica Mar 20 '13 at 21:57
10
  1. Make sure the difference comes from the experimentation not from the adopted technique/method.
  2. Make sure you have the same settings as the other paper. Sometimes people make assumptions for the sake of simplicity in experimentations. For example, I remember I did experimentation assuming acyclic graph exists.
  3. Do you have some kind of randomness (i.e. generating random instances of the problem)? If yes, revise its output. Sometimes you examine easy instances while others base their experimentations on hard instances of the problem.
  4. In some areas, there are benchmarks and robust solvers for particular problems/structures. If your field have benchmarks, try to compare your method against it.

Either way, I am sure you have important parameters to control the experimentation (i.e. number of variables..etc). check their role.

Most importantly, you need to theoretically justify why your method will save 14% while other method saves only 2% in practice.

  • 3
    These are worthy goals, but are not always possible in practice. Unfortunately, the prior paper may not have exposed all the details of its implemented method, nor the precise settings of its experiments, nor the source of its input data. Don't make the same mistakes. The last point is especially difficult; there may be no theoretical justification for why method X works better than method Y in practice, because there is no crisp, accurate, abstract formulation of the phrase "in practice". – JeffE Mar 20 '13 at 15:30
  • @JeffE I totally agree with your comment; that's why I see many evaluating papers in the literature of some fields trying to find which method better than another in practice. – seteropere Mar 20 '13 at 15:46
5

Yes. To improve on others findings is a common situation. The fact that the first paper overstated performance may not necessarily be wrong from th epoint of their experimental setup but they may have missed some component that negatively affected their experiment. I would say that this reflects incremental improveents in the development of ideas in science. As someone once said: "If I knew what I was doing, it wouldn't be science".

5

Some fields deal with exact numbers in which case you don't have a contradiction, you have identified an error. When you are dealing with inexact numbers that have "measurement error", you need to be careful. As much as I dislike statistics, they can be, and really are, your friend when dealing with measurement error.

You say Paper Y found that Method X was 15% better than baseline. Did they do a statistical comparison to see if Method X was better than baseline, or did they calculate confidence intervals and really say that it was 15%+/-0.000001 better than baseline? Is your 2% difference from baseline statistically reliable? Is your 2% difference from baseline statistically different from 15%? Then we have your statements about the modified methods. Is the 14% statistically reliably different from the 2% improvement you saw?

If there is measurement error then all you can say is that it is extremely unlikely that your implementation of their method is the same. This doesn't really contradict them, and it definitely doesn't say they are wrong.

  • 1
    it definitely doesn't say they are wrong — True, but it also definitely erodes any confidence that they are right. – JeffE Mar 20 '13 at 15:33
5

JeffE commented:

It's obvious that it should happen at least 5% of the time, but my impression is that it happens a LOT more often than that.

The 5% aren't all that obvious to me: if I understood correctly, the 5% are the (in)famous p-value.
That is, of every 100 false null-hypothesis, 5 are rejected ("we found something") by mechanically rejecting H0 when the p-value indicates that the probability of observing such or more extreme results as we got reaches 5%.

                        | what the paper does       |
                        | reject H0   not reject H0 | sum
------------------------+---------------------------+------
truth |  null hypotesis |   5                  95   | 100
      v  alternative h. |   ?                   ?   |  ?
------------------------+---------------------------+------
sum                     |   ?                   ?   |          

The number of contradicted papers, however, should depend on the number of falsely accepted hypotheses among all accepted hypotheses (whether true or not). The problem is, we'd need to know the number of correctly accepted alternative hypotheses to calculate which percentage should lead to contradictions.

This we don't know, but of course it depends on the number of true alternative hypotheses among all hypotheses, which we may call the "prevalence of good ideas".
If we stay in analogy to medical terms, the percentage of contradicted papers should be (1 - predictive value of rejected null-hypotheses). And this will be much larger than 5% if lots of "bad" ideas are tested.

Literature:


Here are two comments from pharmaceutical companies reporting on the issue for (mostly oncological) drug development:

  • 2
    p-value, i.e. the probability to falsely reject a true null-hypothesis - This is not the p-value! Instead it is one of its numerous misinterpretations. – Bravo Mar 20 '13 at 22:14
  • @Shyam: please edit to correct me! But note that I'm talking about a (large) number of studies where the null-hypothesis is actually true (100 in my example). (And you should have told me that there was a wrong "accept" just below.) - so please proofread again ;-) – cbeleites supports Monica Mar 20 '13 at 23:55
  • 1
    See also article 1, article2 in Economist. – StasK Oct 17 '13 at 19:12
-2

It is completely commonplace, particularly in finance publications, to fabricate results.

A previous thesis adviser of mine actively encouraged not reporting results which did not support the story he was trying to tell, and to completely change test design and the statistical tests performed when it was possible to get results which did support the story.

It should come as no surprise that papers report results which contradict each other.

  • fabricated results..Really? – seteropere Mar 21 '13 at 11:35
  • 3
    This is a fairly serious charge to bring against an entire field. Do you have anything (non-anecdotal) to support this claim? – eykanal Mar 21 '13 at 12:33
  • Not bringing a claim against an entire field, just mentioning that it is common. Not sure how anyone could deny that... – H. D. Mar 21 '13 at 20:29
  • 5
    Sorry, but "just mentioning" that obviously unethical behavior is "completely commonplace" IS bringing a serious charge against an entire field. – JeffE Mar 22 '13 at 1:43
  • @H.D. Which finance publications you may be talking about? – user13107 Mar 22 '13 at 3:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.