4

I am interested in exporting and redistributing arXiv papers and will need the source on them. It looks like this is allowed so long as they fall under CC BY or CC BY-SA licenses. However, I'm not sure how viable it is, because I don't know how many papers are typically published with those licenses.

It says on their bulk data page that "A small fraction of submissions are made with other licenses and this information is available in the OAI-PMH metadata." So the data are available there, but I'm only not sure how to harvest these data from the OAI.

  • Well you click the link that says OAI-PMH which tells you the protocol version and base url. Then you click the link for the protocol version which explains the protocol so I guess you need to read that and implement the client half of the protocol. Or maybe there is an existing library for it. The arXiv also suggests that one should normally just link to the arXiv for downloads. – Dan Robertson Aug 18 '18 at 20:21
  • @DanRobertson: The protocol description is quite long and complicated, and awkward to implement, so I think it's fair to ask this question and see whether someone has already got this data. – Nate Eldredge Aug 20 '18 at 6:00
8

There are more than a million eprints at arXiv. arXiv started in 1991 well before Creative Commons and Creative Commons licensing existed. The first generation of CC licenses were made public in December 2002.

Therefore, I think to answer this question, it is fair to take just a sample of the (massive!) total eprints, and a recent sample at that - little point in assessing eprints from 1991-2001.

I couldn't work out how to get this information myself from the various arXiv APIs so (full disclosure), I asked Erick Peirson on Twitter, who is currently lead software architect at arXiv, if he could get some data on this matter. The rest of this answer is all his work tbh:

You can use the OAI-PMH endpoint provided by arXiv to query license information.

These are the totals for each submitting-author selected license that arXiv allows for eprints uploaded between 2015-01-01 through to 2018-07-30:

enter image description here

For that period, only 3.42% of eprint uploads choose a license that is not the default arxiv-license 1.0. Of the 13,920 that did not choose the default arxiv-license, 55% (7748) chose either CC BY 3.0 or CC BY 4.0, and only 6.16% (857) used CC BY-SA 3.0.

So to answer you question exactly "What percentage of arXiv preprints fall under the CC BY or CC BY-SA licenses?", of recent eprints it is just over 2% (8605/420567) licensed under those specific licenses.

Thanks for asking such an interesting question. I learned a lot from it!

  • 1
    Thanks! This is quite sad, but it would be nice to make a dump on archive.org with all the freely licensed works (not -NC). – Nemo Sep 3 '18 at 10:31
  • My teammate and I will be sorting through these works (taken from AWS) when we have time in the next 1-2 weeks, I would be happy to provide an AWS link to works separated by license. I'm not sure how to do a dump to archive.org though. – Chylomicron Sep 4 '18 at 20:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.