Why are meta-analyses the most quoted type of research paper despite being often flawed?

Question

Meta-analysis represents the statistical pooling of several individual studies focusing on a specific research question, ideally conducted within the context of a systematic review of the scholarly literature.

Several examples of meta-analysis are available, with the first ones already appearing in the mid '50s, despite this term being proposed first by Gene Glass.

Indeed, it appears that meta-analyses continue to have substantial success in the scholarly literature, and often appear among the most quoted types of research design. Yet, a plethora of meta-analyses of suboptimal quality is being currently published, and this unfortunate trend does not appear to cease.

Which are the main reasons for this success? And, most appropriately, is this scientifically sound and reasonable, given the epidemic of meta-analyses often of suboptimal quality (Ioannidis 2016) or overlapping in content (Biondi-Zoccai et al, BMJ 2006)?

Disclosure: I am the first author of Biondi-Zoccai et al, BMJ 2006, and I have coauthored dozens of meta-analyses.

Answer 1

You have it backwards. Meta-analyses are not heavily cited despite frequently being flawed; rather, they are frequently flawed because meta-analyses are heavily cited.

Because meta-analyses often were high-impact papers, many people with poor training and little experience tried to publish their own meta-analyses, in the hope of boosting their C.Vs. It's (mainly) this flood of incompetent authors who are publishing flawed studies, and dragging the whole field down.

The increase is a consequence of the higher prestige that systematic reviews and meta-analyses have acquired over the years, since they are (justifiably) considered to represent the highest level of evidence. Many scientists now want to do them, leading journals want to publish them, and sponsors and other conflicted stakeholders want to exploit them to promote their products, beliefs, and agendas. Systematic reviews and meta-analyses that are carefully done and that are done by players who do not have conflicts and pre-determined agendas are not a problem, quite the opposite. The problem is that most of them are not carefully done and/or are done with pre-determined agendas on what to find and report.

--We have an epidemic of deeply flawed meta-analyses, says John Ioannidis

Eventually, hopefully, journals and authors will become more aware of this and the poor-quality reviews will be ignored, correcting the problem. As evidence for this, we see a number of guidelines and resources for improving meta-analyses becoming more widely used over the past few years:

• IOM
• PRISMA
• MOOSE

Answer 2

As an epidemiologist, I'll object to the use of "epidemic" in your question - I don't actually think it's justified.

But here are a few reasons meta-analyses are highly cited:

• A single effect estimate is almost meaningless. Multiple studies, in multiple populations, have to be done in order for something to really be taken seriously. Once this is accomplished, a meta-analysis collecting this information becomes the single best source for an overall picture of the literature. This, in turn, makes it easier to cite, discuss, etc.
• A meta-analysis is also an excellent jumping off point for discussions of the state of a given effect, a summary of the current literature, and if done correctly, a means of quantifying and highlighting gaps that exist in our knowledge. That often spawns commentaries, position papers, guidance documents, etc.

Is this scientifically sound? I'd say yes. There are a lot of meta-analysis papers, possibly too many, and like any method, they can be flawed. But they also represent the best way of tackling an entire chunk of a literature at once, and subjecting it to rigorous scrutiny.

Do such an analysis correctly, honestly ask yourself if such an analysis is needed, but if the answer is "Yes", then it's both a valid approach and one likely to be reasonably influential.

Answer 3

Your question is awfully loaded. So first, a few clarifications-

The proper question is not whether or not meta-analyses are flawed. But rather, are meta-analyses more flawed than your average scientific publication? Bear in mind that all forms of scientific publishing are inherently susceptible to falsification. When you're talking about a "flawed" publication you need to define what you mean more precisely. Ostensibly a paper that has gone through peer review and has been accepted should be free of gross defects in methodology. Otherwise, the problem is not really on the paper, but rather on the community that allows poor-quality papers to be published.

Are meta-analyses falsified more often? Are they retracted more often? You haven't said exactly what you mean by low-quality, and so we can only guess at what you mean.

However, I can answer a variant of your question:

Why are statistical meta-analyses useful?

Statistical meta-analysis is useful because single studies are never authoritative. Making authoritative statements requires replication. To see why, let's look at the commonly used p-value. These are used as a way to differentiate the effects of random sampling from true experimental effects between a control and experimental population. In any experiment, there is a chance that an observed effect is purely due to sampling error- suppose you're testing whether or not a chemical causes cancer in mice.

Ideally you'd use enough mice so that minor variations between your experimental and control group don't impact your analysis, but this is not always the case. Even if you use a hundred or a thousand mice, there's some small chance that you were just unlucky and happened to pick a hundred or a thousand mice that were genetically predisposed towards developing cancer. The ex post facto likelihood of this being the case greatly depends on the final difference between the two groups, and this is essentially what is measured by p-values. A smaller p-value is better, and implies there is a smaller chance that the observed experimental difference was due to sampling error.

Now, particulars aside, just know that certain fields of study use the p-value as a minimum barrier to entry for scientific publication. For example, someone might say that for an experiment to be meaningful it must achieve a p-value of less than 0.05. Suppose you run a study that achieves a p-value of 0.045. It's suitable for publication- but having a low p-value doesn't mean that you've got a bulletproof result. All it means is that the probability of your result being skewed by sampling error is "low", but "low" might 50% or higher.

A recent study in psychological science was published that estimates the reproducibility of experimental effects based on p-value. The Minitab Blog interprets this study for us statistical laymen. The basic result is that people sorely overestimate the reliability of experimental results with low p-value. Even good studies with very good p-values (less than 0.001) were not reproducible over 1/3rd of the time. The bare-minimum studies that had a p-value near 0.05 were not reproducible about 2/3rds of the time. The following chart is from the Minitab Blog:

At this point, the answer to my modified question should be clear. Modern science is a statistical endeavor, and (given the statistical tools available to us) individual studies are rarely if ever high-confidence results. Thus, statistical meta-analysis are necessary for making high-confidence claims. This leads to a high citation rate because it isn't the first or second paper on a subject that is authoritative, it is the culmination of several studies that allow researchers to be authoritative.

Answer 4

Because they offer the information prepared in bite-sized pieces. It's the best sources that you can use if you want to generate a paper or thesis quick and dirty.

** Try to understand it from the view of a writer of an usual thesis, paper or article why he/she would use such a meta analysis:**

• If the writer is a student:

  Typical students who are writing thesises are not interested in scrolling through myriads of papers to collect the information they need for writing their work. Students know that their thesises almost matter to no one, so why spending so much effort in collecting pieces from multiple papers/journal entries if you can have something citable much easier?

• If the writer has a job in Academia:

  1. To publish just for publishing:

     If you want to brag with a huge list of your publications, then meta papers are the best way to go. Just recycle their findings as much as possible. IF you can brag with a list of 30 papers, you can leave others merely stunned because they will never invest the effort in order to read all the work and probably see that a part of it is just worthles b.s. that has no added value but only stubborn citation

  2. Monetary reasons:

     It can happen that your (private) donors want to see quick results of their investment. I recently read an article of which the title was like "Why researchers publish papers that no one reads" suggesting that people in academia often publish redundant work in order to pretend that the donors investment shows continuous result (they have to do, because donors often think like Managers, often having little understanding how research works). If you have to really generate such work for coin, I can comprehend that you go the easy way and just recycling findings from meta papers.