I'm looking to do an extensive literature search.

I would like to start with about half a dozen or so survey papers in the field, and look at their bibliography. Then, I would like to go through each of those papers and look at their bibliography.

This can continue until I have, say, 500 - 1000 papers in a a relationship graph, after I've pruned papers that do not contain certain keywords or words in their title/abstract. Then, I can rank the papers based on their connectedness in the graph, weighted by the age of the publication.

Sounds great, but I can't see any easy way to accomplish this. I've looked at DBLP and crossref.org, but neither of these show the bibliography of the papers. Web of Science has an API to which I may be able to get access, but I don't think that will provide bibliography either.

Short of manual web scraping, is there any way to achieve this?

I can't scrape Google Scholar since it doesn't show the bibliography of papers, but Microsoft Academic Search does. MSAS does have an API, but you need to apply with information like Project URL to get access, which is not applicable in my case.


The name for the type of research related to your question is referred to as: Bibliometrics or Scientometrics.

That being said, the answer to your question is not simple since it depends on a few factors. You could be lucky and have a service such as this available that has a lot of the data available for you: http://www.scimagojr.com/journalsearch.php?q=22078&tip=sid&clean=0

There is also a whole host of tools that can help you parse the files without using an API: https://www.mat.unical.it/ianni/wiki/ScholarHIndexCalculator https://sites.google.com/site/hjamali/scientometric-portal

In my case they did not provide all the information I required and ultimately I had to write a few tools of my own. The problem with this approach is that most papers are only available in PDF and requires PDF text extraction. Plenty of frameworks available to assist with that, but the structure of the document needs to be customised for each source of information since they are not all the same. Long story short, the following API's helped me a lot.

A list of scholarly APIs that can assist: http://libguides.mit.edu/apis

Text extraction for individual files: http://sourceforge.net/projects/itextsharp/ https://pdfbox.apache.org/

  • 1
    Thanks for that. I needed a solution reasonably quick, so I threw together some code and scraped MSAS. I've indexed over 7000 papers in my area, shown as a directed graph. I can search the database and identify papers that are influential, etc. – Ozzah Nov 8 '15 at 22:50

Here are some APIs you may want to consider. I found the pybliometrics one to be very useful. https://github.com/pybliometrics-dev/pybliometrics

This one may useful for the network you want to create. https://github.com/shauseth/article-networks

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.