12

My goal is to produce a graph showing the linkages (and lack there of) between several fields that share a common subproblem by showing who cites whom. There are many databases that show citations between papers: ISI Web of Knowledge, Microsoft Academic Research, and Google Scholar. However none allows me to download even part of their database. Is there some database I have overlooked? Has someone written a scrapper for one of these websites?

For reference, the fields I'm considering are Mechanics, Magnetic Resonance Imaging, Signal Processing, Geophysics, and several others.

  • I've always wondered why google scholar doesn't provide the functionality to see "who has this paper cited" in addition to "who has cited this paper". – Andy W Jul 20 '12 at 14:34
5

Our WikiCite project uses Wikidata to store bibliographic information including citation information. The data from Wikidata is free under CC0 license and you are able to access the data via XML dumps, RDF dumps, the web API and the Wikidata Query Service with a SPARQL endpoint.

While the bibliographic data in Wikidata is certainly not complete as of October 2017, Wikidata editors have done considerable work so Wikidata now contains over 8 million scientific articles and over 36 million citations. The coverage may be somewhat ok'ish for a start in Magnetic Resonance Imaging while quite bad in fields such as Mechanics, Signal Processing and Geophysics.

The Wikidata Query Service is used in our Scholia webservice to aggregate and display scholarly information in tables, plots and graphs, including citation graphs. Scholia, via the Wikidata Query Service, allows you to zoom in on a topic and create co-author graphs and co-occuring topics graph with respect to a topic. For instance, for the machine learning concept of "embedding" you can see the graphs here: https://tools.wmflabs.org/scholia/topic/Q29043227. Small partial citation graphs are displayed on the 'work' pages, see, e.g., https://tools.wmflabs.org/scholia/work/Q21090025.

It is possible to extract the bibliographic data with the Wikidata Query Service. For a co-author graph of the big connected component I used a SPARQL query against the Wikidata Query Service and Gephi on the downloaded results. The resulting image - and the SPARQL queries - are available here: https://commons.wikimedia.org/wiki/File:Scholarly_co-author_graph_via_Wikidata,_2017-05-26.png

You can read more about the approach of using Wikidata for scientometrics work in this paper: "Scholia and scientometrics with Wikidata", https://arxiv.org/abs/1703.04222

1

Scraping Google Scholar

I found this:

https://github.com/tonybreyal/Blog-Reference-Functions/blob/master/R/googleScholarXScraper/googleScholarXScraper.R

I also wrote something a while back, though it's not pretty:

https://github.com/ketch/scinet/blob/master/gss.py

Using either of these you could trace citations forward through the literature. Note that Scholar will block you if you hit it too many times, too quickly.


You can download thousands of records at a time from Scopus, if you have access:

  1. Run a search.
  2. Tick the box to select all documents.
  3. Click "create bibliography".
  4. Export to desired format.

However, this doesn't include citation information, so it's probably useless to you.


  • 1
    scraping Google scholar is against the EULA, you might not get caught if you do a little bit, but there are decent consequences if you do get caught (I had a colleague who got his lab in trouble with such an approach) and your librarians can't save you. – Artem Kaznatcheev Jul 21 '12 at 1:45
  • @ArtemKaznatcheev That link leads to a stackoverflow question which has a dead link to (supposedly) the EULA. So I can't find evidence supporting your claim (though I suspect it is true). – David Ketcheson Jul 23 '12 at 12:16
  • My Google-fu has not been sufficient to find a new ToS, this might be due to Google's recent unified ToS approach. I asked a question here. – Artem Kaznatcheev Jul 23 '12 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.