Bulk download of arXiv (or other publication data set) with metadata AND citations

Question

This is to extend the question: Getting a dump of arXiv metadata

I would like to download arXiv in bulk, and found a useful answer above for using this syntax:

http://export.arxiv.org/oai2?verb=ListRecords&set=math&from=2015-01-01&until=2015-01-31&metadataPrefix=arXiv

However, in data there is not a field for citations. I wonder if there is another dataset.

In https://arxiv.org/help/bulk_data there is no description about citations, apart of an old dataset (KDD) of 2003.

Is there any parameter to extend the above query and also fetch citations? Is there any reference to see how arXiv metadata are classified?

Any other dataset of articles possible to freely download? I would like to use a dump for testing visualizations for knowledge discovery. If there is already a sample DB standardized / polished, it will be really helpful not to have to polish / curate data.

Answer 1

arXiv's metadata does not include any citation information.

If you are interested in citations in arXiv documents, your best bet is probably to extract them from the PDF files using a dedicated tool (Cermine, pdfx or pdfextract for instance), or from the LaTeX sources, by inspecting the .aux files.

You can download PDF or source files using the dedicated bulk data interface. Using the OAI interface for this purpose is not recommended as it generates a lot of traffic on the main arXiv site.

You could extract a similar dataset from PubMed Central using their API. If you are looking for something broader (not restricted to a particular topic), you could use COnnecting REpositories (they have full texts from a variety of sources, with an API).

You can also use a pre-curated dataset such as the Microsoft Academic Graph.

Answer 2

Since recently, arXiv citation information is available from Semantic Scholar -- it is the one that is displayed on the arXiv website.

Semantic Scholar apparently provides a dump of their data.