I am working on a systematic literature review and noticed something quite disturbing since I want to base my work on Google Scholar results: The number and actual results returned by Google Scholar seem to depend on the order of the search terms (to some very relevant extent), at least if the
OR operator is used.
This quirk poses a problem, to the extent that it sheds doubt on the reproducibility of the search results. How should I deal with this in my systematic literature review? Is there any plausible explanation for why the results depend on the order of search strings? Knowing this might help to find a workaround. The consequences of this behavior could actually be quite important to any academic user who searches for papers in their own research. Naturally we cannot definitely determine what's going within Google's algorithm as it is a blackbox but we still can come up with hypotheses that we might be able to test nevertheless (I cannot think of anything more scientific than that :))
What I found out so far:
The number of results shown on the top of the results page is known to be a rough estimate that does not necessarily reflect the actual number. Furthermore, only the first ~1000 results/100 result pages are reachable anyway. However, neither is relevant to this question since it happens for far more specific searches with below 10 results as well.
Some examples with their number of results (the results themselves differ too).