0

I want to find a way to bulk-download citation data on (ideally!) all the papers published in my particular academic discipline. I want to analyse this large corpus of data using AI text classification tools, because I want to identify all papers that fit in a certain category. At the very least I want the basic citation info (title, authors, year, journal), and abstracts would be nice to have as well.

Are there tools for scraping the contents of literature databases that could achieve this objective? For example, one approach might be to compile a list of all journals that are relevant in my field, and use that list to download citation data for every single article in each of those journals. Or perhaps downloading citations of all papers that have certain keyword (even if there are very very large numbers of results).

Improve this question
9
  • 1
    I’m sure Web of Science would be happy to cooperate for a fee. But, talk to your local research librarians to get the right way to do it.
    – Jon Custer
    Nov 24, 2022 at 16:06
  • 2
    Could you be a bit more specific about what your actual question is? Is it about how to get the data? Or about about how to analyze the data once you have it?
    – Jochen Glueck
    Nov 24, 2022 at 16:44
  • 3
    @John Custer a hefty fee.
    – henning
    Nov 24, 2022 at 16:46
  • 2
    @henning - well, yes, but you get an actual dataset. And don’t violate various user agreements.
    – Jon Custer
    Nov 24, 2022 at 18:21
  • 2
    They are not exactly copyrighted, it is not a blanket ban on obtaining this data. You would not breach copyright law in this case, but ToS - very likely, as this aggregated data is at the very core of Clarivate's and the like business models. They did the hard part making sense of the bibliometric stuff with wildly different styles used in the text, and now you just want to just download it in a digestible, clean format. Not so fast.
    – Lodinn
    Nov 25, 2022 at 12:18

3 Answers 3

Reset to default
2

First, get a corpus of all the DOIs that constitute the 'field' you are interested in. (One possible way to get the DOIs would be, first, to obtain all the relevant journals' ISSNs and then to find out all their DOIs in CrossRef's API. Here is an example.)

Second, filter these DOIs in the COCI (OpenCitations Index of Crossref open DOI-to-DOI citations) and fetch the DOIs that are cited by the former (citing) DOIs. Here is an example.

Third, use each (cited) DOI and loop them through CrossRef's API so as to obtain the data you need (title, authors, year, and journal will certainly be available, in some cases even the Abstracts). Here is an example.

Improve this answer
2
  • This seems like a really promising approach. Two questions: 1. Do you know of an efficient API-friendly way to bulk-find ISSN numbers based on a list of journal names? 2. How can I return all the DOIs for each journal using CrossRef's API? The example you gave only returns 20 results.
    – Roger
    Dec 2, 2022 at 16:27
  • 1
    (1.) Hm, not really... I sometimes used Web of Science's Master Journal List (or Scopus' source coverage lists) for finding the ISSN of journals of a given discipline, but that will be limited to the respective database's coverage. (2.) Oh, you can show more results by adding &rows=100 or so. But there's probably a limit, so you have to use paging/cursors. Look at the CrossRef API documentation of how cursors work: github.com/CrossRef/rest-api-doc
    – anpami
    Dec 2, 2022 at 18:47
6

There may be tools that can do this on a small scale, but you will find that most publishers will block your attempts to download data in bulk using scripts. For example, ACM's usage policies of their websites explicitly prohibit the use of scripts. I happen to know because of a colleague's attempt at understanding what a subset of ~1000 articles in their journal I co-edit provides: The publisher promptly blocked his IP address from accessing any more article pages.

The issue at hand is not that citation data is copyrighted (it may or may not -- that's immaterial here), but that you are accessing a service that costs publishers money (namely, their websites) and there are service conditions attached to using their service. As a consequence, unless you have a specific agreement with publishers, you will find that trawling large numbers of websites is not going to lead to much success.

Improve this answer
2
  • Not sure if it costs publishers much money, but it certainly makes them money (+1).
    – henning
    Nov 26, 2022 at 21:52
  • @henning How much it actually costs them is not the right question. It does cost them something, and so it is not unreasonable that they can set the conditions for your access to their service.
    – Wolfgang Bangerth
    Nov 27, 2022 at 20:02
1

For this sort of work I strongly recommend looking into Dimensions.ai which is run by Digital Science. Dimensions is specifically designed to do what your looking for, and if you can get access to their API you will be able to efficiently pull and analyse large quantities of metadata/publications across entire fields. Ideally your institution will already have access to Dimensions and can help you out with an API key.

As for other options, it varies by database. Some subject specific databases will let you download bulk metadata but will often have an upper limit on bulk downloads (a few hundred for example).

I know Google Scholar does not like you pulling bulk metadata. If you are wanting to stick with Google Scholar your best bet is to use a plug-in like Zotero, which will let you quickly capture all the metadata (and potentially readily available .pdf's) of results on a page-by-page basis. If you start pulling too much metadata too quickly from Google Scholar you will trip their bot detection captcha's.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .