3

I would like to download a database containing all scientific papers published. This is obviously too broad and ambitious. To be realistic, say I want all the papers searchable on google scholar (although other databases are fine, say SemanticScholar, etc.), or an approximation of that. What databases out there allow you to download this data?

I don't want to download the papers themselves (that wouldn't fit on my storage), just a record containing title, authors, journal (if it is not a book), year, and perhaps abstract.

Is this possible? How can I download a database like this, that I can browse and search offline?

Improve this question
5
  • 5
    No, it’s not possible.
    – Dan Bron
    Commented Oct 22, 2017 at 20:01
  • 3
    Get a database admin job at Thomson Reuters?
    – Nate Eldredge
    Commented Oct 22, 2017 at 20:05
  • @DanBron That's pessimistic. A way to do it is to wget google scholar searches with a single letter, a, b, c, ... A simple script could do it. I am just wondering if there is an easier way to do it.
    – a06e
    Commented Oct 22, 2017 at 21:40
  • 4
    At best that would get you all papers indexed by Google Scholar. But Google Scholar is not an index of all scientific papers; at best, it's an index of scientific papers whose metadata is available online. (Similarly, not all scientific papers are indexed by Thomson Reuters, or even by the union of Google Scholar and Thomson Reuters.)
    – JeffE
    Commented Oct 23, 2017 at 0:23
  • @JeffE I should have been more specific in my question. But an index containing all of Google Scholar or an approximation of it, is more than sufficient for me.
    – a06e
    Commented Oct 23, 2017 at 1:04

2 Answers 2

Reset to default
8

Probably the closest you can get is the Open Academic Graph, a graph-based database of about 321 million papers (I don't think the actual papers are stored, but all the information you want is there). You can download all the 143 GB of data from the above link, but do note that because it's a merge of two existing graph-based paper databases (Microsoft Academic Search and AMiner) there are going to be a lot of duplicates.

Improve this answer
2
  • Thanks (+1). Do you know if the data from Microsoft Academic Search can be downloaded by itself?
    – a06e
    Commented Oct 23, 2017 at 1:08
  • 1
    @becko Yes, the downloads in the link separate the data of both databases. You'd want just the MAS ones in that case.
    – Crazymoomin
    Commented Oct 23, 2017 at 8:49
6

You can download a number of databases, but none, as far as I know, are comprehensive across all fields.

You might also want to check the list of repositories compliant with the OAI-PMH metadata harvesting protocol.

Improve this answer
0

Not the answer you're looking for? Browse other questions tagged .