40

Prompted by discussion in this post on Meta.MathOverflow.net I got interested in comparing usage of tags from MathOverflow to submissions in the respective disciplines of arXiv. (Vide a similar idea of language popularity, GitHub vs StackOverflow (or one from 2015).) Moreover, as people often use their real names on MO, it may be interesting to check the overlap of mathematicians.

The question is, how to get such data?

There is arXiv API, but for bulk downloads of metadata they recommend Open Archives Initiative (OAI). Yet, as I see, it can query one article at a time, and one needs to know its id. So without knowing arXiv ids beforehand, it turns into a guessing game.

There are some plots in arXiv usage statistics, yet I don't see this exact data.

Also, one can get total submission to math from links in Mathematics -> Article statistics by year, but it misses the splitting into subdisciplines.

6
  • Looking at the OAI, it looks like there's also a cap in place to prevent a high amount of leeching (503 after a certain period).
    – Compass
    Feb 16, 2015 at 16:39
  • @Compass In any case, do you know what are the relevant queries? Feb 16, 2015 at 16:41
  • I have been playing with this a little bit, using the Perl package Net::OAI::Harvester. The OAI API does provide a way to download all records (ListRecords), rather than just one at a time (GetRecord). It looks like arXiv will give you 1000 records and then return a 503 asking you to retry after a delay, so it's not to prevent excessive leeching as @Compass says but only to rate limit. Net::OAI::Harvester knows to how to retry (listAllRecords) but not how to delay, so that may need to be done manually. Feb 16, 2015 at 17:58
  • 4
    @Mark : you might try asking the question on opendata.stackexchange.com . (disclaimer : I'm currently a moderator there)
    – Joe
    Feb 18, 2015 at 2:05
  • 1

4 Answers 4

3

Arxiv metadata and fulltext have been made (more easily) accessible in 08/2020.

The full set of PDFs is available for free in the GCS bucket gs://arxiv-dataset or through Google API (json documentation and xml documentation, gsutil). They are grouped into several .tar.gz files in the tarpdfs folder, the complete set is about 1.1TB in size.

1
  • 1
    So something good is coming out of machine learning at last! (Though I'm a bit surprised to see the PDFs rather than the source codes in the dump. And wow -- the metadata alone is over 4GB??) Aug 5, 2020 at 15:27
23

My main confusion was not realizing that The Open Archives Initiative Protocol for Metadata Harvesting is a separate protocol, not a subset of arXiv API.

In this case, the relevant queries are ListIdentifiers (10k items per query) and ListRecords (1k items per query). To get just identifiers we need to write:

http://export.arxiv.org/oai2?verb=ListIdentifiers&set=math&metadataPrefix=oai_dc

It results in 10k identifiers in the following form:

<OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd">
  <responseDate>2015-02-16T19:28:22Z</responseDate>
  <request verb="ListIdentifiers" metadataPrefix="oai_dc" set="math">http://export.arxiv.org/oai2</request>
  <ListIdentifiers>
    <header>
      <identifier>oai:arXiv.org:0704.0002</identifier>
      <datestamp>2008-12-13</datestamp>
      <setSpec>math</setSpec>
    </header>
    ...
    <header>
      <identifier>oai:arXiv.org:0712.1769</identifier>
      <datestamp>2011-06-23</datestamp>
      <setSpec>math</setSpec>
    </header>
    <resumptionToken cursor="0" completeListSize="249546">760571|10001</resumptionToken>
  </ListIdentifiers>
</OAI-PMH>

As there are more results, to get next batch we need to specify resumptionToken, in this case:

http://export.arxiv.org/oai2?
   verb=ListIdentifiers&resumptionToken=760571|10001

and so on.

Other useful parameters are from and until, e.g. as in

http://export.arxiv.org/oai2?verb=ListIdentifiers&set=math&metadataPrefix=oai_dc&from=2015-01-14&until=2015-01-14

To directly get categories (bear in mind that set=math specifies mathematics, but there are no smaller subsets), one can write:

http://export.arxiv.org/oai2?verb=ListRecords&set=math&from=2015-01-01&until=2015-01-31&metadataPrefix=arXiv

It's important to set metadataPrefix=arXiv, so that subdisciplines will be listed:

<categories>
  math-ph cond-mat.other math.MP nlin.CD physics.class-ph
</categories>

EDIT:

I used delay as Nate Eldredge suggested, in my case - 25s. Yet, while trying to get all math (250k items so in 250 queries) it gave error at 70. I did continue it (with even higher delay) but sometime around 110 the query was not longer available.

So, the way to go is in getting smaller chunks - e.g. by month (or for mathematics - at most by year).

2
  • 3
    When making multiple requests using a resumptionToken, you are required to wait a little while between requests; otherwise you get a 503 error with a Retry-After: header telling you how long to wait. In the tests I have made so far it has always been 20 seconds, so adding an unconditional 20 second delay between requests might do the trick. Feb 16, 2015 at 20:25
  • 3
    This looks good. Here you can see the code we are using to get metadata. It is fetching only on page of it though for now.
    – Mitar
    Feb 16, 2015 at 23:03
17

Shameless plug: I wrote a generic OAI harvesting tool, that will harvest Arxiv just fine. It's called metha and consists of a few commands:

$ metha-sync https://export.arxiv.org/oai2

This will download all data up to the last full day (it will take a couple of days). The XML API responses are compressed and placed under ~/.cache/metha directory. Metha will use monthly windows by default and a resilient HTTP client to ensure downloads succeed, while not stressing the server. It has been tested in the wild on thousands of OAI endpoints.

After (and during) download, you can inspect (already downloaded) records with:

$ metha-cat https://export.arxiv.org/oai2

For any further processing you will have to use your favorite XML tools.


Update: Additionally to the metha (incremental) harvester, I wrote a small tool called oaicrawl, which does no caching and just fetches records off an OAI endpoint one by one. This create more overhead, as there's an HTTP request for each record but can be useful, if the OAI endpoint does not support selective harvesting (e.g. by date) or is otherwise broken and you are ok with having a best effort data set harvested from the service.

Syntax would be similarly simple:

$ oaicrawl http://export.arxiv.org/oai2 > arxiv.data

Note, that this will concatenate the raw responses from the API and hence won't be valid XML out of the box.

4

A torrent for a metadata dump "collected from the OAI-PMH API endpoint using the 'metha-sync' tool" is available at: https://archive.org/details/arxiv-bulk-metadata

NB: This dataset contains metadata also for non-math articles.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .