Export Google Scholar search for fine-grained analysis

Question

Problem

I recently started to survey an algorithm described in a paper. I would like to include some variants of this algorithm in my survey.

One natural choice is to use "cited by" feature in Google Scholar. It could return a new search of the papers that cite the previous one (see below).

Since there are 1342 papers, it is not feasible to read all of them (even just abstracts) within short period of time. I made a heuristic for selecting "interesting" papers

Only read the papers that have more than some number of citations (say K).

which means I will only have to read papers that in turn have more than K "cited by" number.

Even though this heuristic is unfair for newly published papers, I believe it could indeed help me narrow down my scope.

However, the issue is Google Scholar does not have features to support my heuristic (something like advanced search supported by other databases might). So my problem is

• How do I download the search results from my Google Scholar query with all the meta data, including "cited by" information.
• If the previous point is not possible, is there another tool that could help me?

Could someone help me. Thank you in advance!

Answer 1

There is no API for google scholar data. They do that on purpose ... Here are links to some possible workarounds

• https://pypi.org/project/scholarly/

• https://www.nature.com/articles/d41586-018-04190-5

• https://serpapi.com/google-scholar-api

Answer 2

You can scrape Google Scholar Search Results using BeautifulSoup web scraping library.

Check code in online IDE:

from bs4 import BeautifulSoup
import requests
import re
import json


headers = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.53 Safari/537.36"
}

params = {
    "hl": "en",     # language of the search
    "q": "covid",   # search query
    "num": 100,     # number of results per page. In this case, 100 results per page
}

response = requests.get(
    "https://scholar.google.com/scholar", params=params, headers=headers
)

soup = BeautifulSoup(response.content, "lxml")

organic_results_data = []

for organic_result in soup.select(".gs_r.gs_or.gs_scl"):
    title = organic_result.select_one(".gs_rt a").text
    snippet = organic_result.select_one(".gs_rs").text

    cited_by_match = re.search(
        r"Cited by (?P<cited_by_count>\d+)",
        organic_result.select_one(".gs_or_btn.gs_nph+ a").text,
    )
    cited_by_count = int(cited_by_match.groupdict().get("cited_by_count"))

    organic_results_data.append(
        {"title": title, "snippet": snippet, "cited_by_count": cited_by_count}
    )

print(json.dumps(organic_results_data, indent=2))

Example output:

[
  {
    "title": "Thoracic imaging tests for the diagnosis of COVID‐19",
    "snippet": "Background The respiratory illness caused by SARS‐CoV‐2 infection continues to present \ndiagnostic challenges. Our 2020 edition of this review showed thoracic (chest) imaging to be …",
    "cited_by_count": 186
  },
  {
    "title": "An overview of COVID-19",
    "snippet": "… 11, 2020, the World Health Organization (WHO) officially named the disease resulting from \ninfection with SARS-CoV-2 as coronavirus disease 2019 (COVID-19). COVID-19 represents …",
    "cited_by_count": 333
  },
  {
    "title": "COVID-19 and multiorgan response",
    "snippet": "… COVID-19 has demonstrated a wide spectrum of clinical … multiorgan impact of COVID-19 \nreported since its outbreak. … If a paper is reporting on many aspects of the COVID-19, then the …",
    "cited_by_count": 729
  },
  # ...
]

---

Alternatively, you can use the free open-source package scholarly.

Example from its documentation.

>>> search_query = scholarly.search_keyword('Haptics')
>>> scholarly.pprint(next(search_query))
{'affiliation': 'Postdoctoral research assistant, University of Bremen',
 'citedby': 56666,
 'email_domain': '@collision-detection.com',
 'filled': False,
 'interests': ['Computer Graphics',
               'Collision Detection',
               'Haptics',
               'Geometric Data Structures'],
 'name': 'Rene Weller',
 'scholar_id': 'lHrs3Y4AAAAJ',
 'source': 'SEARCH_AUTHOR_SNIPPETS',
 'url_picture': 'https://scholar.google.com/citations?view_op=medium_photo&user=lHrs3Y4AAAAJ'}