Historical accuracy of trends/number of publications on keywords using Google Scholar?

Question

Following this question on SE Academia: Easy way to see historical trend in keywords in publications?, I'm using Google Scholar and the linked script (https://github.com/Pold87/academic-keyword-occurrence) to get the number of publications on a certain keyword.

For the record, I was using the tool to get some statistical data for the popularity of the keyword "inter(-)organis(z)ational trust" (multiple spellings) (this should not matter for the the actual question though).

I found that there are very few to no publications at all on this topic between the 1940s and 1985. I'm wondering if this is (was) actually the case OR if this is just because there's no data on Google Scholar that dates back to that era, i.e. there were indeed publications on that topic but none of them got into Google Scholar because no-one cares about those publications now.

TL;DR

So my question is whether you think the part of publications before 1985 that is not on Google Scholar is really "negligible" and the numbers on Google Scholar are representative (historically accurate) for the period of time between 1940 and today.

FYI, I attached a plot of the numbers from GS:

EDIT:

If not GS, do you think the data on Constellate (https://constellate.org) is attendible (or is there literally any service you know of that fits this purpose)?

Answer 1

Google Scholar is essentially a byproduct of Google Search. It picks up everything in the web crawl that looks like a scholarly paper, and indexes it. There isn't much manual curation, and unlike most publications databases it doesn't start by working from a list of publications it thinks it should have.

(If you are Scopus, you index the Journal of X by starting at Volume 1 and recording all the papers. If you are Google Scholar, you do it by collating all the papers from X that turned up in the search, relying on them being online to avoid gaps. It does then deduce the existence of other papers it finds in the citations - but only by title/author etc.)

This means that Google Scholar is very good at picking up modern material that is on the web, wherever it might be hosted, but very bad at identifying material that is not on the web, or only referred to in a static list somewhere. More and more older material is trickling online, ending up in the big waves of digitisations or on repositories or wherever... but there's still a long way to go. And of course, a lot of the older material is not well-digitised - it may be in flat non-OCRed PDFs, if at all. So it will be invisible to in-text keyword searching, which is what you're wanting to do.

It's striking that your numbers start to rise properly in the mid-1990s. This is the point at which journals started to routinely publish online, and it becomes more likely there would be a full-text readable version for the crawler to pick up. Prior to that, you're maybe going to be trying to match your keywords against just the title, or maybe an abstract at best - so of course you get fewer hits compared to the period when you can match full text.

(It might be worth benchmarking your search against some other terms - run it for "deflation" or "deforestation" or "fishing rights" or whatever. I have a suspicion you'll see a similar sort of curve...)

Lastly, you ask about Constellate. This is derived from the old JSTOR Data for Research service, and is mostly made up from records in JSTOR and Portico; this means there will be some very substantial gaps depending on which publishers they have signed up to their service. That is likely to skew the results one way or another, but whether it's going to be an issue for your topic, I don't know.

Answer 2

No they are not attendibile at all. Google Scholar focuses on living researchers and on readily accessibile online publications or metadata therein.

Older publications may not exist in digital format, or they may have poor ocr performed (an example is the “t” being recognized as “f” … it was happening in the early 00s and 10s on many publications from the 70s through the 80s).