I am working on a systematic literature review and noticed something quite disturbing since I want to base my work on Google Scholar results: The number and actual results returned by Google Scholar seem to depend on the order of the search terms (to some very relevant extent), at least if the OR operator is used.

This quirk poses a problem, to the extent that it sheds doubt on the reproducibility of the search results. How should I deal with this in my systematic literature review? Is there any plausible explanation for why the results depend on the order of search strings? Knowing this might help to find a workaround. The consequences of this behavior could actually be quite important to any academic user who searches for papers in their own research. Naturally we cannot definitely determine what's going within Google's algorithm as it is a blackbox but we still can come up with hypotheses that we might be able to test nevertheless (I cannot think of anything more scientific than that :))

What I found out so far:

The number of results shown on the top of the results page is known to be a rough estimate that does not necessarily reflect the actual number. Furthermore, only the first ~1000 results/100 result pages are reachable anyway. However, neither is relevant to this question since it happens for far more specific searches with below 10 results as well.

Some examples with their number of results (the results themselves differ too).

  • 2
    Ask Google about their search algoritms - not on topic here. – Solar Mike Jun 9 at 5:06
  • 2
    The question could easily be changed into an on-topic one by asking "How should I deal with this in my SLR", instead of "What's going on here". – lighthouse keeper Jun 9 at 7:14
  • 1
    The question is interesting for most academics: many of them use Google Scholar, and may not be aware of this problem or its solution. – Louic Jun 9 at 7:46
  • 2
    I've slightly reworded and reorganized your question to highlight why it is on-topic here, in line with what @lighthousekeeper suggested. – henning -- reinstate Monica Jun 9 at 11:55
  • 1
    @louic I disagree. He's following the systematic literature review (SLR) methodology, in which it's indeed important to have a systematic coverage of the papers one is looking for. This issue is entirely orthogonal to the subject/topic of the SLR. – lighthouse keeper Jun 9 at 15:10

If you want to use google scholar as a means of measuring the number of papers on a certain topic: don't.

Contrary to what we often see in reviews or other publications, you did some more research than simply copying the number of results from a naïve search, and that research indicated that this number is highly unreliable (as should have been expected). Conclusion: Google scholar is not the right tool for the job. Its algorithms are unknown, and the sources it searches are unknown as well. Maybe it even counts the same paper multiple times, depending on which words you search for, who knows?

Having said that, in practice nobody really seems to mind an introduction stating "a Google search for XXX returned N results", but the question remains how useful such a statement is, given your findings.

| improve this answer | |
  • I am not (really) interested in the number of results. You are totally right in what you write - and yes, Google counts some documents twice sometimes if they differ only slightly. This is all known and not topic of my question though. I am primarily interested in the results themselves. I am scraping them and will analyze them further by looking at better sources for the metadata. However, for that it is important that I discover the papers I am looking for and as I have shown it is not obvious what one gets from GS queries that look to me like they should result in the same output. – stefanct Jun 9 at 11:11
  • 1
    @stefanct Perhaps it would be useful to include this specific aim in your question. – henning -- reinstate Monica Jun 9 at 11:57
  • @henning--reinstateMonica maybe, but I tried to formulate it rather generally. after all this behavior of GS is widely unknown AFAICT but relevant to every academic using it. the answer above is focusing on the number issue and i have rephrased the question already to better explain that this is not the point/problem. your edit has hopefully dealt with the problems discussed in the question's comment section - thank you! – stefanct Jun 9 at 12:20

Not the answer you're looking for? Browse other questions tagged or ask your own question.