Research has shown that most of the published studies in various fields are false. For example, the Reproducibility Project: Psychology only managed to confirm 36% of previous 100 psychology studies had a statistically signifiant result the second time. Also, only 6 of 53 studies considered landmark in cancer were successfully replicated. In these circumstances, when such a quantity of false studies pass the peer-review process in most scientific fields, how can we filter authentic information from fields we don't have expertise in?
-
2Welcome to Academia SE and thank you for your question. Can you please edit it to clarify whether you want to ask about p hacking (as indicated by your title) or general problems leading to irreproducibility (as described in the body of your question). I recommend the former, as the latter would be rather broad.– Wrzlprmft ♦Commented Dec 27, 2016 at 16:40
-
5From (RP confirmed 36% of results) does not follow that (64% of results are false).– Dmitry SavostyanovCommented Dec 27, 2016 at 19:48
Add a comment
|
2 Answers
First thing to consider is that research can never provide absolute certainty. Any result you find is only preliminary. You can have various degrees of preliminary: a single study with an acceptable but not brilliant design can provide interesting clues or first steps for a larger project, but not much more. It is right and productive that these studies are published, but don't treat them as truth.
Second thing to consider is that the problem is not necessarily with the articles themselves, but with the lack of replication. So, if you find a result that interest you, try to find replications. If none exist, then you need to be more careful.
answered Dec 27, 2016 at 16:39
-
Your first point is very important. p = 0.05 always means there is a recognized possibility that the result will not replicate even if the research was done and analyzed as perfectly as possible.. However, the OP's invocation of "p-hacking" indicates that they are concerned with papers whose findings are not robust to replication b/c their p-values are not accurate due to design or analyses that compromise a valid interpretation of the published p-value.. Commented Dec 28, 2016 at 18:31
The p-curve (see p-curve.com) has been proposed as a way of identifying whether a set of results is likely to have been p-hacked. The term and technique was coined and/or developed by Uri Simonsohn and colleagues, who blog about replication and p-curving at Data Colada.
- "P-Curve Won’t Do Your Laundry, But Will Identify Replicable Findings"
- "Ambitious P-Hacking and P-Curve 4.0"
- "P-curve vs. Excessive Significance Tests"
The technique has been applied by several research groups besides Simonsohn et al, ie Head et al 2015
There have been critics of p-curves (including John "most published finding are wrong" Ionnadis ), as well as other techniques such as excessive significance tests and the distribution of published p-values.
Some references:
- Simonsohn, Simmons, Nelson (2014) "P-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results" Perspectives on Psychological Science V9(6) p.666-681
- Simonsohn, Nelson, Simmons, (2014) "P-curve: A Key to the File Drawer," Journal of Experimental Psychology: General, V143(2), p.534-547
- Simonsohn, Simmons, Nelson (2015) "Better p-curves" Journal of Experimental Psychology: General.
- Ioannidis and Bruns. 2016. p-Curve and p-Hacking in Observational Research. PLoS ONE.
- Head et al 2015 The Extent and Consequences of P-Hacking in Science. PLoS ONE
