What percentage of arXiv preprints fall under the CC BY or CC BY-SA licenses?

Question

I am interested in exporting and redistributing arXiv papers and will need the source on them. It looks like this is allowed so long as they fall under CC BY or CC BY-SA licenses. However, I'm not sure how viable it is, because I don't know how many papers are typically published with those licenses.

It says on their bulk data page that "A small fraction of submissions are made with other licenses and this information is available in the OAI-PMH metadata." So the data are available there, but I'm only not sure how to harvest these data from the OAI.

Answer 1

There are more than a million eprints at arXiv. arXiv started in 1991 well before Creative Commons and Creative Commons licensing existed. The first generation of CC licenses were made public in December 2002.

Therefore, I think to answer this question, it is fair to take just a sample of the (massive!) total eprints, and a recent sample at that - little point in assessing eprints from 1991-2001.

I couldn't work out how to get this information myself from the various arXiv APIs so (full disclosure), I asked Erick Peirson on Twitter, who is currently lead software architect at arXiv, if he could get some data on this matter. The rest of this answer is all his work tbh:

You can use the OAI-PMH endpoint provided by arXiv to query license information.

These are the totals for each submitting-author selected license that arXiv allows for eprints uploaded between 2015-01-01 through to 2018-07-30:

• http://arxiv.org/licenses/nonexclusive-distrib/1.0/: 406647,
• http://creativecommons.org/licenses/by-nc-sa/3.0/: 485,
• http://creativecommons.org/licenses/by-nc-sa/4.0/: 3187,
• http://creativecommons.org/licenses/by-sa/4.0/: 857,
• http://creativecommons.org/licenses/by/3.0/: 766,
• http://creativecommons.org/licenses/by/4.0/: 6982,
• http://creativecommons.org/licenses/publicdomain/: 192,
• http://creativecommons.org/publicdomain/zero/1.0/: 1451

For that period, only 3.42% of eprint uploads choose a license that is not the default arxiv-license 1.0. Of the 13,920 that did not choose the default arxiv-license, 55% (7748) chose either CC BY 3.0 or CC BY 4.0, and only 6.16% (857) used CC BY-SA 3.0.

So to answer you question exactly "What percentage of arXiv preprints fall under the CC BY or CC BY-SA licenses?", of recent eprints it is just over 2% (8605/420567) licensed under those specific licenses.

Thanks for asking such an interesting question. I learned a lot from it!