1

I am conducting a systematic review with meta-analysis soon and wondering, is there a systematic way to decide on which databases and other sources (publications and grey literature) to search?

It seems to me that the overall effect estimate will depend upon which studies are included in the review. Included studies are determined by inclusion criteria, but - before that - by the choice of which databases to search in the first place.

At the moment I have identified around 30 databases/sources that I intend to search, these include the main publication databases (PubMed, Embase, etc) and also grey literature (preprints and clinical trial registries).

So, I am confident that my search strategy is relatively comprehensive in terms of which sources I search. However, I am not sure that somebody addressing the same PICO question as I am would choose the same sources. Therefore, we are likely to obtain different overall results to the same question, due to some arbitrary decision about which sources to search.

Is there any objective or systematic way of choosing sources? How would you decide, in a way that is not merely arbitrary?

Improve this question
2
  • Are you familiar with the Cochrane Collaboration? If yes, have you reviewed their Handbook? Chapter 4 is a good start. training.cochrane.org/handbook/current/chapter-04
    – Diana Petitti
    Dec 25, 2021 at 18:15
  • Thank you, I checked section 4.3 of Cochrane, the collaboration advises that databases chosen ought to be as wide as possible, and cites some of the major sources. However, Cochrane do not provide any kind of an algorithm for deciding on which databases to actually include (and which to exclude). At least I now know what the current best practice in the field is. Thank you, best wishes.
    – Asymptotic_Tri
    Jan 10, 2022 at 17:36

1 Answer 1

Reset to default
2

It might help to understand that a systematic review is not a mathematical or algorithmic procedure with a precise and perfect result. That is, even though the goal is that two different review teams should reach the same results, because a systematic review is a complex social process, it is impossible for two independent teams to arrive at exactly the same result, no matter how systematic they are. For example, even if both teams follow exactly identical processes (which is practically impossible), but one team searches in December 2021 and the other team searches in January 2022, the results would not be identical.

If you come to terms with this reality, then you should be able to accept that the goal of a systematic review is not to be perfectly reproducible but to be as reproducible as is reasonably expected by scholars who do high-quality professional work.

Regarding the selection of databases, it is reasonable to limit your search to the best-known and most widely used scholarly databases for the specific discipline or disciplines that fall within the scope of the review. These can be determined by reading past reviews from the relevant disciplines and being sure to at least include all the databases that these have used. (Of course, your own review team might not have access to all the databases that you find in the literature, so this might not be feasible.) If two separate review projects strive for that standard, even if they do not use identical databases, I estimate that they should have 95% or greater overlap in studies that they locate. (The greatest disparity would be with grey literature. For example, if both teams search Google Scholar and get 200,000 results on the keyword search, if both teams are maniacally meticulous, one team might look up only 500 results whereas the other might look up 1000, so results would not be identical.)

However, if both teams target the same databases that are known by the discipline as the most comprehensive, most of the highest quality peer-reviewed results should be found in both teams' searches. The main differences should be in work that is published in dates between the two searches, including grey literature.

So, in brief:

  • There is no truly non-arbitrary way to choose literature databases. There will always be arbitrary choices made.
  • Professional quality means doing your best by following past precedent to include the databases that are widely acknowledged to be the most relevant for the discipline.
  • Systematicity does not mean being perfectly reproducible, which is impossible--if that is what it meant, then no one is "systematic". It means being professional to be as systematic as can be reasonably expected.

One more thing: a closely related theme to systematicity is transparency, which means being explicit to report all procedures. This is important because, given that perfect systematicity is impossible, transparency allows future researchers to better understand why their different choices might result in slightly different results from yours.

Improve this answer
2
  • Thank you for your considered reply. It is reassuring to know that I am not expected to achieve impossible standards of thoroughness, and that 'reasonably comprehensive' is considered acceptable by the scientific community. However, I do still find it kind of disconcerting that even the Cochrane collaboration do not provide an algorithm for deciding which sources to search. One study can make all the difference between a meta-analysis favoring control and a meta-analysis favoring intervention. If that study is included or excluded arbitrarily, the overall result will be different.
    – Asymptotic_Tri
    Jan 10, 2022 at 17:43
  • I can imagine a future in which AI is deployed to systematically and comprehensively source all relevant articles from all relevant sources, extract the information and perform meta-analysis. It would be a lot faster and more comprehensive than humans are capable of. Best wishes, Asymptotic Tri
    – Asymptotic_Tri
    Jan 10, 2022 at 17:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .