2

I would like to download a database containing all scientific papers published. This is obviously too broad and ambitious. To be realistic, say I want all the papers searchable on google scholar (although other databases are fine, say SemanticScholar, etc.), or an approximation of that. What databases out there allow you to download this data?

I don't want to download the papers themselves (that wouldn't fit on my storage), just a record containing title, authors, journal (if it is not a book), year, and perhaps abstract.

Is this possible? How can I download a database like this, that I can browse and search offline?

closed as off-topic by Brian Tompsett - 汤莱恩, scaaahu, Coder, Buzz, Florian D'Souza Oct 23 '17 at 14:17

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • ""Shopping" questions, which seek recommendations or lists of individual universities, academic programs, publishers, journals, research topics or similar as an answer or seek an assessment or comparison of such, are off-topic here. (See this discussion for more information.)" – Brian Tompsett - 汤莱恩, scaaahu, Coder, Buzz, Florian D'Souza
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 4
    No, it’s not possible. – Dan Bron Oct 22 '17 at 20:01
  • 3
    Get a database admin job at Thomson Reuters? – Nate Eldredge Oct 22 '17 at 20:05
  • @DanBron That's pessimistic. A way to do it is to wget google scholar searches with a single letter, a, b, c, ... A simple script could do it. I am just wondering if there is an easier way to do it. – becko Oct 22 '17 at 21:40
  • 3
    At best that would get you all papers indexed by Google Scholar. But Google Scholar is not an index of all scientific papers; at best, it's an index of scientific papers whose metadata is available online. (Similarly, not all scientific papers are indexed by Thomson Reuters, or even by the union of Google Scholar and Thomson Reuters.) – JeffE Oct 23 '17 at 0:23
  • @JeffE I should have been more specific in my question. But an index containing all of Google Scholar or an approximation of it, is more than sufficient for me. – becko Oct 23 '17 at 1:04
7

Probably the closest you can get is the Open Academic Graph, a graph-based database of about 321 million papers (I don't think the actual papers are stored, but all the information you want is there). You can download all the 143 GB of data from the above link, but do note that because it's a merge of two existing graph-based paper databases (Microsoft Academic Search and AMiner) there are going to be a lot of duplicates.

  • Thanks (+1). Do you know if the data from Microsoft Academic Search can be downloaded by itself? – becko Oct 23 '17 at 1:08
  • 1
    @becko Yes, the downloads in the link separate the data of both databases. You'd want just the MAS ones in that case. – Crazymoomin Oct 23 '17 at 8:49
5

You can download a number of databases, but none, as far as I know, are comprehensive across all fields.

You might also want to check the list of repositories compliant with the OAI-PMH metadata harvesting protocol.

Not the answer you're looking for? Browse other questions tagged or ask your own question.