36

Prompted by discussion in this post on Meta.MathOverflow.net I got interested in comparing usage of tags from MathOverflow to submissions in the respective disciplines of arXiv. (Vide a similar idea of language popularity, GitHub vs StackOverflow (or one from 2015).) Moreover, as people often use their real names on MO, it may be interesting to check the overlap of mathematicians.

The question is, how to get such data?

There is arXiv API, but for bulk downloads of metadata they recommend Open Archives Initiative (OAI). Yet, as I see, it can query one article at a time, and one needs to know its id. So without knowing arXiv ids beforehand, it turns into a guessing game.

There are some plots in arXiv usage statistics, yet I don't see this exact data.

Also, one can get total submission to math from links in Mathematics -> Article statistics by year, but it misses the splitting into subdisciplines.

  • Looking at the OAI, it looks like there's also a cap in place to prevent a high amount of leeching (503 after a certain period). – Compass Feb 16 '15 at 16:39
  • @Compass In any case, do you know what are the relevant queries? – Piotr Migdal Feb 16 '15 at 16:41
  • I have been playing with this a little bit, using the Perl package Net::OAI::Harvester. The OAI API does provide a way to download all records (ListRecords), rather than just one at a time (GetRecord). It looks like arXiv will give you 1000 records and then return a 503 asking you to retry after a delay, so it's not to prevent excessive leeching as @Compass says but only to rate limit. Net::OAI::Harvester knows to how to retry (listAllRecords) but not how to delay, so that may need to be done manually. – Nate Eldredge Feb 16 '15 at 17:58
  • 4
    @Mark : you might try asking the question on opendata.stackexchange.com . (disclaimer : I'm currently a moderator there) – Joe Feb 18 '15 at 2:05
  • 1
20

My main confusion was not realizing that The Open Archives Initiative Protocol for Metadata Harvesting is a separate protocol, not a subset of arXiv API.

In this case, the relevant queries are ListIdentifiers (10k items per query) and ListRecords (1k items per query). To get just identifiers we need to write:

http://export.arxiv.org/oai2?verb=ListIdentifiers&set=math&metadataPrefix=oai_dc

It results in 10k identifiers in the following form:

<OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd">
  <responseDate>2015-02-16T19:28:22Z</responseDate>
  <request verb="ListIdentifiers" metadataPrefix="oai_dc" set="math">http://export.arxiv.org/oai2</request>
  <ListIdentifiers>
    <header>
      <identifier>oai:arXiv.org:0704.0002</identifier>
      <datestamp>2008-12-13</datestamp>
      <setSpec>math</setSpec>
    </header>
    ...
    <header>
      <identifier>oai:arXiv.org:0712.1769</identifier>
      <datestamp>2011-06-23</datestamp>
      <setSpec>math</setSpec>
    </header>
    <resumptionToken cursor="0" completeListSize="249546">760571|10001</resumptionToken>
  </ListIdentifiers>
</OAI-PMH>

As there are more results, to get next batch we need to specify resumptionToken, in this case:

http://export.arxiv.org/oai2?
   verb=ListIdentifiers&resumptionToken=760571|10001

and so on.

Other useful parameters are from and until, e.g. as in

http://export.arxiv.org/oai2?verb=ListIdentifiers&set=math&metadataPrefix=oai_dc&from=2015-01-14&until=2015-01-14

To directly get categories (bear in mind that set=math specifies mathematics, but there are no smaller subsets), one can write:

http://export.arxiv.org/oai2?verb=ListRecords&set=math&from=2015-01-01&until=2015-01-31&metadataPrefix=arXiv

It's important to set metadataPrefix=arXiv, so that subdisciplines will be listed:

<categories>
  math-ph cond-mat.other math.MP nlin.CD physics.class-ph
</categories>

EDIT:

I used delay as Nate Eldredge suggested, in my case - 25s. Yet, while trying to get all math (250k items so in 250 queries) it gave error at 70. I did continue it (with even higher delay) but sometime around 110 the query was not longer available.

So, the way to go is in getting smaller chunks - e.g. by month (or for mathematics - at most by year).

  • 3
    When making multiple requests using a resumptionToken, you are required to wait a little while between requests; otherwise you get a 503 error with a Retry-After: header telling you how long to wait. In the tests I have made so far it has always been 20 seconds, so adding an unconditional 20 second delay between requests might do the trick. – Nate Eldredge Feb 16 '15 at 20:25
  • 3
    This looks good. Here you can see the code we are using to get metadata. It is fetching only on page of it though for now. – Mitar Feb 16 '15 at 23:03
15

Shameless plug: I wrote a generic OAI harvesting tool, that will harvest Arxiv just fine. It's called metha and consists of a few commands:

$ metha-sync http://export.arxiv.org/oai2

This will download all data up to the last full day (it will take a couple of days). The XML API responses are compressed and placed under ~/.metha directory. Metha will use monthly windows and a resilient HTTP client to ensure downloads succeed, while not stressing the server. It has been tested in the wild on hundreds of OAI endpoints.

After (and during) download, you can inspect (already downloaded) records with:

$ metha-cat http://export.arxiv.org/oai2

For any further processing you will have to use your favorite XML tools.


Update: Additionally to the metha (incremental) harvester, I wrote a small tool called oaicrawl, which does no caching and just fetches records off an OAI endpoint one by one. This create more overhead, as there's an HTTP request for each record but can be useful, if the OAI endpoint does not support selective harvesting (e.g. by date) or is otherwise broken and you are ok with having a best effort data set harvested from the service.

Syntax would be similarly simple:

$ oaicrawl http://export.arxiv.org/oai2 > arxiv.data

Note, that this will concatenate the raw responses from the API and hence won't be valid XML out of the box.

2

A torrent for a metadata dump "collected from the OAI-PMH API endpoint using the 'metha-sync' tool" is available at: https://archive.org/details/arxiv-bulk-metadata

NB: This dataset contains metadata also for non-math articles.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.