2

I'd like to perform a meta-analysis on the efficacy of web-based counsellor interventions across different modalities (text counselling, video counselling, etc.)

To do so I created the following search term:

(Internet OR Online OR Web-based) AND (Therapy OR Counselling OR Intervention) AND (RCT OR Randomized Controlled Trial) AND (Counsellor OR Therapist OR Psychologist OR Psychotherapist)

In Google Scholar this returns over 145 000 results, and still over 19 000 if I filter for 2013 and later.

This is my first attempt at a meta-analysis and I'm wondering if this is considered normal - if there are researchers who sift through tens of thousands of papers to find <100 relevant ones? It seems like other reviews have managed to tackle similar topics yet only returned ~1000 studies. For example: Guided Internet‐based vs. face‐to‐face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta‐analysis

  • 2
    The study you linked says that they used PubMed mainly, which is far less inclusive than Google Scholar. They say they "consulted" more broad databases like Google Scholar, but that sounds very intentionally vague. Most work of this sort starts more narrow for precisely the reason you note, so you might want to start there and see if there are less total results, and of course include less junk. Then you can add other sources, but starting with a massive pile of general junk requires a different approach (usually automation assisted). – BrianH Apr 17 '18 at 21:07
  • if you are comfortable with writing a little bit of code, you could automate some of the process. Either by directly searching for relevant terms within the HTML versions of the article or downloading the pdf's and converting to text and then parsing them. You would still need to check the articles by hand but you could use an algorithm to filter them to a more manageable level. – JWH2006 Jun 27 '18 at 12:24
2

The short answer to the last question is that yes, for some questions people do sift through tens of thousands of results to find a small yield of relevant studies. (And the pool becomes smaller when later screening for details like N and standard errors and adequate description of the population and intervention to allow for a technical meta-analysis.) But that may not be the ideal way for you to do this.

The Campbell Collaboration helps coordinate literature reviews and metaanalyses on many social issues, as the Cochrane Collaboration does for health. The Campbell Collaboration has a document on how to write a good protocol for such a review, including specifying your search strategy, and there's some longer guidance on search strategies and information retrieval here.

The latter document says:

The decision as to how much to invest in the search process depends on the question a review addresses and the resources that are available. It should be noted, however, that article abstracts identified through a literature search can be ‘scan-read’ very quickly to ascertain potential relevance. The process can be facilitated by answering a few specific questions that relate to the inclusion criteria when scanning the abstracts, such as “Does this document report an empirical study?” or “Are children age 3-18 in the sample?”. If the answer to any question is No, then the reference is excluded. But any study that makes it through all of the questions answered either Yes or Can’t Tell, is retrieved. At an estimated reading rate of two abstracts per minute, the results of a database search can be ‘scan-read’ at the rate of 120 per hour (or approximately 1000 over an 8-hour period), so the high yield and low precision associated with systematic review searching is not as daunting as it might at first appear in comparison with the total time to be invested in the review. (p. 23)

Much of the work done by these collaborations involves large research groups and/or volunteer labor and/or automated screening. For your workflow, you would probably want to find a system that lets you view the title and full abstract without clicking through--easier in some databases, maybe a deep option somewhere in Google Scholar. This difference in the view may allow you to more efficiently screen the literature even if there are many potential results.

It seems to me that a good reason for using Google Scholar would be if you want to seek out "gray literature," technical reports, unpublished results, and results in multiple languages. However, Google Scholar will probably bring up more irrelevant studies than if you search databases. For your search terms, though, I'm not sure that you'll run into as many well-designed (RCT) studies on a novel technical infrastructure that would fall through the cracks like that.

Thus, I think your best bet might be doing your main search via PsycINFO or some other database, plus a dissertation database, and then look at Google Scholar a bit to see if anything was systematically missed.

1

I won't repeat what cactus_pardner has already said in their excellent answer (I especially appreciate the abstract-sifting strategy quoted from the Campbell Collaboration), but I will rather focus on the proper (or improper) use of Google Scholar.

Google Scholar is a great resource in general (I consult it regularly myself), but it is not a good primary database for a literature review. For that kind of purpose, searching Google Scholar is similar to doing a regular Web search on regular Google: you will come up with hundreds or thousands of irrelevant articles. Of course, Google is well qualified to build a professional scholarly database, but as far as I understand, in order to allow Google index their articles for free, the publishers place many restrictions on what Google Scholar is allowed to display. For example, Google Scholar is forbidden to display abstracts of articles (which is one of the most crucial capabilities that you need to do a literature review!).

Thus, I recommend that you book an appointment with a librarian to understand the scholarly databases that you have access to (an academic librarian is preferred); a librarian should be able to help you identify the best databases that you have access to that are relevant to your research topic. (And in my experience, librarians can often give you access to special trial databases that are not available to the general public.)

Google Scholar is mainly useful to find non-published grey literature as a supplement to your formal database searches, but as a primary search source, its abundant results of primarily irrelevant results are often overwhelming to the point of being useless.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.