Getting a dump of arXiv metadata

Question

Prompted by discussion in this post on Meta.MathOverflow.net I got interested in comparing usage of tags from MathOverflow to submissions in the respective disciplines of arXiv. (Vide a similar idea of language popularity, GitHub vs StackOverflow (or one from 2015).) Moreover, as people often use their real names on MO, it may be interesting to check the overlap of mathematicians.

The question is, how to get such data?

There is arXiv API, but for bulk downloads of metadata they recommend Open Archives Initiative (OAI). Yet, as I see, it can query one article at a time, and one needs to know its id. So without knowing arXiv ids beforehand, it turns into a guessing game.

There are some plots in arXiv usage statistics, yet I don't see this exact data.

Also, one can get total submission to math from links in Mathematics -> Article statistics by year, but it misses the splitting into subdisciplines.

Answer 1

Arxiv metadata and fulltext have been made (more easily) accessible in 08/2020.

• https://blogs.cornell.edu/arxiv/2020/08/05/leveraging-machine-learning-to-fuel-new-discoveries-with-the-arxiv-dataset/
• https://www.kaggle.com/Cornell-University/arxiv

The full set of PDFs is available for free in the GCS bucket gs://arxiv-dataset or through Google API (json documentation and xml documentation, gsutil). They are grouped into several .tar.gz files in the tarpdfs folder, the complete set is about 1.1TB in size.

Answer 2

My main confusion was not realizing that The Open Archives Initiative Protocol for Metadata Harvesting is a separate protocol, not a subset of arXiv API.

In this case, the relevant queries are ListIdentifiers (10k items per query) and ListRecords (1k items per query). To get just identifiers we need to write:

http://export.arxiv.org/oai2?verb=ListIdentifiers&set=math&metadataPrefix=oai_dc

It results in 10k identifiers in the following form:

<OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd">
  <responseDate>2015-02-16T19:28:22Z</responseDate>
  <request verb="ListIdentifiers" metadataPrefix="oai_dc" set="math">http://export.arxiv.org/oai2</request>
  <ListIdentifiers>
    <header>
      <identifier>oai:arXiv.org:0704.0002</identifier>
      <datestamp>2008-12-13</datestamp>
      <setSpec>math</setSpec>
    </header>
    ...
    <header>
      <identifier>oai:arXiv.org:0712.1769</identifier>
      <datestamp>2011-06-23</datestamp>
      <setSpec>math</setSpec>
    </header>
    <resumptionToken cursor="0" completeListSize="249546">760571|10001</resumptionToken>
  </ListIdentifiers>
</OAI-PMH>

As there are more results, to get next batch we need to specify resumptionToken, in this case:

http://export.arxiv.org/oai2?
   verb=ListIdentifiers&resumptionToken=760571|10001

and so on.

Other useful parameters are from and until, e.g. as in

http://export.arxiv.org/oai2?verb=ListIdentifiers&set=math&metadataPrefix=oai_dc&from=2015-01-14&until=2015-01-14

To directly get categories (bear in mind that set=math specifies mathematics, but there are no smaller subsets), one can write:

http://export.arxiv.org/oai2?verb=ListRecords&set=math&from=2015-01-01&until=2015-01-31&metadataPrefix=arXiv

It's important to set metadataPrefix=arXiv, so that subdisciplines will be listed:

<categories>
  math-ph cond-mat.other math.MP nlin.CD physics.class-ph
</categories>

EDIT:

I used delay as Nate Eldredge suggested, in my case - 25s. Yet, while trying to get all math (250k items so in 250 queries) it gave error at 70. I did continue it (with even higher delay) but sometime around 110 the query was not longer available.

So, the way to go is in getting smaller chunks - e.g. by month (or for mathematics - at most by year).

Answer 3

Shameless plug: I wrote a generic OAI harvesting tool, that will harvest Arxiv just fine. It's called metha and consists of a few commands:

$ metha-sync https://export.arxiv.org/oai2

This will download all data up to the last full day (it will take a couple of days). The XML API responses are compressed and placed under ~/.cache/metha directory. Metha will use monthly windows by default and a resilient HTTP client to ensure downloads succeed, while not stressing the server. It has been tested in the wild on thousands of OAI endpoints.

After (and during) download, you can inspect (already downloaded) records with:

$ metha-cat https://export.arxiv.org/oai2

For any further processing you will have to use your favorite XML tools.

---

Update: Additionally to the metha (incremental) harvester, I wrote a small tool called oaicrawl, which does no caching and just fetches records off an OAI endpoint one by one. This create more overhead, as there's an HTTP request for each record but can be useful, if the OAI endpoint does not support selective harvesting (e.g. by date) or is otherwise broken and you are ok with having a best effort data set harvested from the service.

Syntax would be similarly simple:

$ oaicrawl http://export.arxiv.org/oai2 > arxiv.data

Note, that this will concatenate the raw responses from the API and hence won't be valid XML out of the box.

Answer 4

A torrent for a metadata dump "collected from the OAI-PMH API endpoint using the 'metha-sync' tool" is available at: https://archive.org/details/arxiv-bulk-metadata

NB: This dataset contains metadata also for non-math articles.