Automatically building a database of forward and backward citations

Question

My goal is to produce a graph showing the linkages (and lack there of) between several fields that share a common subproblem by showing who cites whom. There are many databases that show citations between papers: ISI Web of Knowledge, Microsoft Academic Research, and Google Scholar. However none allows me to download even part of their database. Is there some database I have overlooked? Has someone written a scrapper for one of these websites?

For reference, the fields I'm considering are Mechanics, Magnetic Resonance Imaging, Signal Processing, Geophysics, and several others.

Answer 1

Scraping Google Scholar

I found this:

https://github.com/tonybreyal/Blog-Reference-Functions/blob/master/R/googleScholarXScraper/googleScholarXScraper.R

I also wrote something a while back, though it's not pretty:

https://github.com/ketch/scinet/blob/master/gss.py

Using either of these you could trace citations forward through the literature. Note that Scholar will block you if you hit it too many times, too quickly.

---

You can download thousands of records at a time from Scopus, if you have access:

1. Run a search.
2. Tick the box to select all documents.
3. Click "create bibliography".
4. Export to desired format.

However, this doesn't include citation information, so it's probably useless to you.

---