5

Like so many others, I scrape data from Google Scholar as a part of my lit review process, so that I can have a structured data set for meta-analysis of the literature.

I noticed that for a couple of many topics of interest, the # of articles per year seems to be increasing until 2017, then it drops off sharply.

I wonder if it's really safe to assume that fewer articles were published in 2018?

Is it possible that this means that the data through 2017 is relatively "complete", whereas journals and authors for 2018 may still be in the process of being added to Google's index, thus the total number is under-reported?

Has anyone encountered this?

  • maybe you could check and compare this observation with other search index's like scopus and ISI web of knowledge. Also arxiv.org has a statistics section to check this academia.stackexchange.com/a/126982/41661 – user48953094 May 13 at 17:49
  • @user847982 I like that idea, though I guess there's a hurdle of needing to write a separate data scraper for those. I was also wondering if I could try using the same scraper with Google Scholar and different topics, to see if I see this happening across the board within Google Scholar – Hack-R May 13 at 18:36
7

Google Scholar has its strong points (e.g. indexing of grey literature that is not available in any regular scholarly database), but data quality is not one of them. Of course, this is not because Google lacks the ability to create a high quality database; it is rather because publishers refuse to grant it permission to create a high-quality database that it distributes for free. Google's index is based on Google Scholar's web spider whose completeness depends on what is available from public websites (Google strictly respects websites permissions; it makes no attempt to index anything where the websites ask it not to do so with a robots.txt entry). I would not be surprised if some publishers restrict Google's permission to index details of some of their most recent publications.

With that perspective, then for any given topic, if there is a sharp dropoff during or after 2017 (its unclear which is the case the way you worded the question), I would not consider that evidence of anything. That is, it is not necessarily evidence that people suddenly stopped publishing on that topic; it is only evidence that Google's index no longer contains that topic, for whatever reason. I know that I've seen quite a few articles that have charts like that and make claims like that, but I don't consider such claims reliable. (And when I peer-review articles that make such claims, I tell the authors so.)

To make any concrete, serious claim about change in publishing patterns of topics, you would need a more rigorous and systematic database source (such as Web of Knowledge, Scopus, etc.) and at least a two-year lag to make sure that all data is complete.

  • 1
    Thank you very much (+1). I agree with you on the 2 year lag and much of what you said. Are you sure that Google Scholar creates its create index in the same what that regular Google search does? It would be good to have a link to something discussing that. I was under the impression that they combined self-reported results from the journals and authors with some special/different data scrapers/connectors built for the journal sites. That was just my perception though, so I may well be wrong. Either way, the 2-year lag is a good idea. How do we know that WoK or Scopus are better? – Hack-R May 13 at 21:25
  • 1
    @Hack-R as a meta-analyst myself, I would say to be careful with Google Scholar, especially regarding it's limitation (no wildcards, limits of characters in search). From my experience, WoK has way more possibilities. You should check published meta-analysis in your field and note where did they do their systematic search. – Emilie May 14 at 12:10
  • @Emilie Thank you as well; that is useful. Based on your recommendation and the one in this answer, I will write a data scraper for it today. – Hack-R May 14 at 14:43
  • @Hack-R, about WoK or Scopus being better, I don't have any citations to give you, but I have colleagues who research bibliometrics and for them it's not even a question. Even though the Clarivate (WoK) and Elsevier (Scopus) teams have already done a lot of cleanup from what the journals give them, bibliometricians still need to do a lot of their own cleanup. In contrast, Google Scholar is hopelessly unusable for precise scientometric purposes, to paraphrase one of my bibliometrician colleagues. It's great for general searches, but not very useful for scientific bibliometrics. – Tripartio May 14 at 22:53
  • @Hack-R, no, I'm pretty certain that Google Scholar uses custom spiders different from those of the regular Google Web search; however, it is all Google technology. (Please note I said "spiders", not "scrapers".) I'm editing my answer to make this more explicit. – Tripartio May 14 at 22:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.