How to account for publication bias?

Question

I am doing a literature review in the medical field, which is as many other fields heavily impacted by publication bias. Is there any standard way to account for it?

When am I e.g. allowed to say that treatment X successfully treat condition Y, despite the fact that a certain amount of studies may have been left unpublished as they did not show statistically significant "positive" results, i.e. that those unpublished studies did not successfully show that treatment X does treat condition Y?

Or shall I just ignore it?

Answer 1

First, you should note the difference between checking for evidence of publication bias and correcting for it. To know that there are patterns in the data that indicates publication bias means that you will be able to temper the discussion of your results, and maybe suggest potential weaknessess in the study, even if it is not possible to correct for biases.

There exists a number of methods that can be used to identify publication bias. A simple graphical funnel plot of effect size vs. sample size is maybe the easiest to apply. The idea here is that the scatter around the true effect size should be symmetric and with expanding variance at smaller sample sizes ("trumpet-shaped"). Publication bias can lead to underreporting of non-significant studies with smaller sample sizes, which would result in a skewed pattern in the funnel plot. Another method to test for publication bias is a regression between standardized effects and precision ('Egger's method') or rank-correlation methods. You can get an overview of these methods and others in e.g. Macaskill et al. (2001) and Thornton & Lee (2000).

Accounting/correcting for publication bias is trickier, and I have no personal experiance in using these methods. However, I know that one method that is used in medical research is to only include pre-registred studies in the meta-analysis. If this is possible depends on your topic though. You can also use simulations and parametric methods to basically try to recreate the complete (unknown) dataset, and in this way account for the publication bias (one example of this is Givens et al. 1997)

Answer 2

This is a very good question. Unfortunately, I don't think that there is (or can be) a "standard" answer. Consider: if you found 10 papers finding a strong treatment effect of X on Y, you a priori have no way of knowing whether these 10 studies were all studies on the X-Y relationship... or whether there were 20 studies, 10 of which were published, while 10 others showed no relationship and ended up in a file drawer... or whether in fact there were 100 studies, 90 of which repose peacefully in file drawers all over the world.

That said, there are a few standard ways of at least investigating the structure of published effects, among them the funnel plot. However, the funnel plot and similar methods will usually only be good for investigating publication bias, not for accounting for it. And I would argue, as per the previous paragraph, that if you try to account for publication bias in any quantitative way, you need to be extremely careful not to overstate your certainty in this accounting, given the uncertainty about inputs into your accounting.

Anyway, while a question like this is certainly on-topic here, you may actually get more answers at CrossValidated, although a quick search didn't turn up a lot. You may want to consider posting an analogous question there. Two potentially enlightening questions are here and here, the last of which has a very good answer pointing to a book, from which I'll quote just one sentence: "Good meta-analyses endeavour to obtain unpublished studies."

EDIT: finally, you may want to check out and/or contact the Bias Methods Group of the Cochrane Collaboration, especially this list of references.

Answer 3

In general, it may be a question in statistics rather than 'academia'. E.g. by looking at the distribution of p-values you may have an educated guess about acceptance distribution.

See this Economist article for description of the problem and this post/paper on the distribution of p-values in psychology (in short: too many just a bit below .05).

Answer 4

As mentioned above, there are a number of statistical methods available. The funnel plot is probably the oldest, but it has its "issues" (Lau, J., et al (2006). BMJ (Clinical Research Ed), 333(7568), 597–600)

An interesting, newer method is Ioannidis's test for excess significance (Ioannidis, J. P. A., & Trikalinos, T. A. (2007). Clinical Trials (London, England), 4(3), 245–253).

However, all these techniques amount to "statistical divining rods", telling you whether the results seem too good to be true.

If you really want to determine whether publication bias is present, you need to know (a) the results (not just the existence) of unpublished studies and (b) whether the results in the published literature have undergone "statistical alchemy" and, if so, what the true "unspun" results are.

Since you say you're working in medicine, you have a unique opportunity, at least if you're working with drugs. You can use FDA drug approval packages, as I have in my own work. you may find these papers of interest:

Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. The New England Journal of Medicine, 358(3), 252–260. doi:10.1056/NEJMsa065779

Turner, E. H., Knoepflmacher, D., & Shapley, L. (2012). Publication Bias in Antipsychotic Trials: An Analysis of Efficacy Comparing the Published Literature to the US Food and Drug Administration Database. PLoS Medicine, 9(3), e1001189. doi:10.1371/journal.pmed.1001189

Turner, E. H. (2013a). How to access and process FDA drug approval packages for use in research. BMJ (Clinical Research Ed), 347(oct14 2), f5992–f5992. doi:10.1136/bmj.f5992

Hope this helps.