8

Research has shown that most of the published studies in various fields are false. For example, the Reproducibility Project: Psychology only managed to confirm 36% of previous 100 psychology studies had a statistically signifiant result the second time. Also, only 6 of 53 studies considered landmark in cancer were successfully replicated. In these circumstances, when such a quantity of false studies pass the peer-review process in most scientific fields, how can we filter authentic information from fields we don't have expertise in?

  • 2
    Welcome to Academia SE and thank you for your question. Can you please edit it to clarify whether you want to ask about p hacking (as indicated by your title) or general problems leading to irreproducibility (as described in the body of your question). I recommend the former, as the latter would be rather broad. – Wrzlprmft Dec 27 '16 at 16:40
  • 5
    From (RP confirmed 36% of results) does not follow that (64% of results are false). – Dmitry Savostyanov Dec 27 '16 at 19:48
8

First thing to consider is that research can never provide absolute certainty. Any result you find is only preliminary. You can have various degrees of preliminary: a single study with an acceptable but not brilliant design can provide interesting clues or first steps for a larger project, but not much more. It is right and productive that these studies are published, but don't treat them as truth.

Second thing to consider is that the problem is not necessarily with the articles themselves, but with the lack of replication. So, if you find a result that interest you, try to find replications. If none exist, then you need to be more careful.

| improve this answer | |
  • Your first point is very important. p = 0.05 always means there is a recognized possibility that the result will not replicate even if the research was done and analyzed as perfectly as possible.. However, the OP's invocation of "p-hacking" indicates that they are concerned with papers whose findings are not robust to replication b/c their p-values are not accurate due to design or analyses that compromise a valid interpretation of the published p-value.. – N Brouwer Dec 28 '16 at 18:31
4

The p-curve (see p-curve.com) has been proposed as a way of identifying whether a set of results is likely to have been p-hacked. The term and technique was coined and/or developed by Uri Simonsohn and colleagues, who blog about replication and p-curving at Data Colada.

The technique has been applied by several research groups besides Simonsohn et al, ie Head et al 2015

There have been critics of p-curves (including John "most published finding are wrong" Ionnadis ), as well as other techniques such as excessive significance tests and the distribution of published p-values.

Some references:

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.