9

This is to extend the question: Getting a dump of arXiv metadata

I would like to download arXiv in bulk, and found a useful answer above for using this syntax:

http://export.arxiv.org/oai2?verb=ListRecords&set=math&from=2015-01-01&until=2015-01-31&metadataPrefix=arXiv

However, in data there is not a field for citations. I wonder if there is another dataset.

In https://arxiv.org/help/bulk_data there is no description about citations, apart of an old dataset (KDD) of 2003.

Is there any parameter to extend the above query and also fetch citations? Is there any reference to see how arXiv metadata are classified?

Any other dataset of articles possible to freely download? I would like to use a dump for testing visualizations for knowledge discovery. If there is already a sample DB standardized / polished, it will be really helpful not to have to polish / curate data.

  • This answer might be related to the second part of your question (getting citation counts to articles), but it does not allow for bulk downloads of citations. – fileunderwater Nov 29 '15 at 22:23
  • 1
    What do you mean by "see how arxiv metadata are classified"? – pintoch Nov 29 '15 at 22:52
8

arXiv's metadata does not include any citation information.

If you are interested in citations in arXiv documents, your best bet is probably to extract them from the PDF files using a dedicated tool (Cermine, pdfx or pdfextract for instance), or from the LaTeX sources, by inspecting the .aux files.

You can download PDF or source files using the dedicated bulk data interface. Using the OAI interface for this purpose is not recommended as it generates a lot of traffic on the main arXiv site.

You could extract a similar dataset from PubMed Central using their API. If you are looking for something broader (not restricted to a particular topic), you could use COnnecting REpositories (they have full texts from a variety of sources, with an API).

You can also use a pre-curated dataset such as the Microsoft Academic Graph.

  • Thank you for CORE - did not know that. Are you saying that to extract metadata of citations one need to parse pdf by pdf ? It would be quite of a work for standardizing all the elements and typos and such. I thought citations may have been already cleaned by publishers. Is it not available publicly in bulk (e.g. at least, one can download patents corpus from google in xML, but at least XML not pdf!) – user305883 Dec 1 '15 at 18:42
  • If you are looking for a clean dataset of citations, have a look at the last sentence of my answer :-) – pintoch Dec 1 '15 at 22:18
4

Since recently, arXiv citation information is available from Semantic Scholar -- it is the one that is displayed on the arXiv website.

Semantic Scholar apparently provides a dump of their data.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.