Google scholar is an excellent service for looking at basic metadata about publications (e.g. authors, title, abstract, and papers it cites). But they do not offer an official API (free or otherwise).

Is there also a non-commercial database of such publications? I'm wondering because this is something that perhaps libraries in universities would require to organize information about publications across disciplines.

Since there are non-profit systems like doi.org and cross-ref which let publications get an identification numbers, I'm wondering if all disciplines use them. Or is it the case that universities get this metadata from a variety of publishers (like Springer, Elsevier, etc) and there is no single central index?

To avoid bikeshedding on what a "major discipline" is, we could limit ourselves to what's already found on google scholar, or perhaps to what top 100 (or choose your number) university libraries in the world might index.

  • Google Scholar may not have an "official" API, but it does accept queries in a reasonably consistent URL syntax, and returns results in reasonably consistently-formatted HTML, so it should be possible to script access to it. I took a look in Google Scholar's terms of use, and couldn't see any clauses that obviously forbid scripting. Dec 9 '20 at 14:31
  • 1
    @DanielHatton unfortunately such approaches are always extremely labor-intensive (because the page structure may update) or incomplete (it may not be possible to easily extract information added by javascript to the page), and on top of that subject to rate limits. Dec 10 '20 at 2:16

I'm going to suggest https://ui.adsabs.harvard.edu/ Although, it does not index everything it does index many astronomy, physics, maths, engineering, and other science journals (Full list of journals http://adsabs.harvard.edu/abs_doc/journals1.html). There are even some "arts" journals though they are usually combined arts and science journals.

It also has an api https://github.com/adsabs/adsabs-dev-api


LENS.org offers a free API comprising data from ca. 224 million scholarly work (according to that page about the API).

In comparison, CrossRef has ca. 120 million records (source), also with an API.

But note that the metadata from both LENS and CrossRef are limited. For instance, they do not track self-cites, they do not analyse author collaboration networks, they do not generate author profiles through algorithms (though LENS seems to plan doing so with the help of ORCiD). For such advanced purposes, commercial databases such as Web of Science, Scopus, or Dimensions are the go-to platforms.

  • 1
    CrossRef is mentioned in the question ...
    – Ben Bolker
    Dec 6 '20 at 21:33
  • @BenBolker that's alright, but I do not know if crossref is considered to have a good coverage across disciplines. If there is consensus about it having good coverage, then that's a sufficient answer. This answer seems to affirm that this is the case. Dec 7 '20 at 10:30
  • I mentioned CrossRef again because I was convinced that it was the most comprehensive free database of scholarly publications with an API. - However, I did a research now and found out that LENS actually had a greater coverage and an API (last time I checked, many months ago, it did not provide an API). I thus edited my response. (And thank you, Peeyush!)
    – anpami
    Dec 7 '20 at 12:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.