What is the most effective and efficient method of searching multiple journals for multiple keywords?

Question

My goal is to find articles related to invasive species management on a monthly basis. I have a list of species for my region (i.e keywords) and a list of journals. Although I enjoy combing through new issues of each journal, I'm wondering if there is a more automated, and less time consuming method.

I've tried using Boolean operators in Google Scholar, but this limits the search to 2019 (not monthly) and isn't journal specific. I have also explored some programs (e.g. Stork), with limited results.

Any insight is greatly appreciated. Thanks!

Answer 1

In case the journals are indexed by PubMed ( biomedical literature) and you're not afraid to do a bit of programming, you might want to explore the tools proposed here. You can even download the full Medline database (see here) and search through it locally, in case that's more convenient.

Answer 2

As far as I know you can only limit Google Scholar searches by year, not by month. This might still be fine for a specific enough search, as you can just tell from a glance what's new or not*. To keep it specific, you might want to do several search queries for different keywords instead of chaining the keywords together using Boolean operators. What will really help you, however, is to use the source qualifier to limit results to a specific journal. You can also search in several specified journals using a query like

"invasive species management" OR "biological invasion" (source:"ecology letters" OR source:"ecological economics")

In case the journals you're interested in have names that overlap with other journals, see this question.

*If not, you could presumably automate the queries, and compare the results to a cached version from the previous run. Would certainly be more involved than just a search query though.

Answer 3

If you know on which website the journal is located:

"invasive species management" OR "biological invasion" + site:eprints.gla.ac.uk

To get exact result use + instead of OR: <keyword> + <keyword> + <source> OR <source>:

"invasive species management" + "biological invasion" + source:"ecology letters"

---

If you are okay with the programmatic approach, you can use Python and Google Scholar Organic Results API from SerpApi. Check out the playground.

It's a paid API with a free plan that bypasses blocks from Google and does all the hard lifting parts so the end-user only needs to think about what data to extract.

SerpApi also supports Ruby, Node.js, Go, PHP, Java, Dotnet, and Google Spreadsheets.

Code and example in the online IDE to extract data from all pages:

import os, json
from serpapi import GoogleSearch
from urllib.parse import urlsplit, parse_qsl

params = {
    # os.getenv(): https://docs.python.org/3/library/os.html#os.getenv
    "api_key": os.getenv("API_KEY"),  # Your Serpapi API key
    "engine": "google_scholar",       # search engine
    #  search query
    "q": '"invasive species management" OR "biological invasion" source:"ecology letters" OR source:"ecological economics"',
    "hl": "en",                       # language
    # "as_ylo": "2017",               # from 2017
    # "as_yhi": "2021",               # to 2021
    "start": "0"                      # first page
    }

search = GoogleSearch(params)         # where data extraction happens

organic_results_data = []

papers_is_present = True
while papers_is_present:
    results = search.get_dict()      # JSON -> Python dictionary

    print(f"Currently extracting page №{results.get('serpapi_pagination', {}).get('current')}..")

    for result in results["organic_results"]:
        position = result["position"]
        title = result["title"]
        publication_info_summary = result["publication_info"]["summary"]
        result_id = result["result_id"]
        link = result.get("link")
        result_type = result.get("type")
        snippet = result.get("snippet")

        organic_results_data.append({
            "page_number": results.get("serpapi_pagination", {}).get("current"),
            "position": position + 1,
            "result_type": result_type,
            "title": title,
            "link": link,
            "result_id": result_id,
            "publication_info_summary": publication_info_summary,
            "snippet": snippet,
            })

        if "next" in results.get("serpapi_pagination", {}):
            search.params_dict.update(dict(parse_qsl(urlsplit(results["serpapi_pagination"]["next"]).query)))
        else:
            papers_is_present = False

print(json.dumps(organic_results_data, indent=2, ensure_ascii=False))

Output:

[
  {
    "page_number": 1,
    "position": 1,
    "result_type": null,
    "title": "Evidence of climatic niche shift during biological invasion",
    "link": "https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1461-0248.2007.01060.x",
    "result_id": "UXttzv_h5ScJ",
    "publication_info_summary": "O Broennimann, UA Treier, H Müller‐Schärer… - Ecology …, 2007 - Wiley Online Library",
    "snippet": "… Our results report, for the first time, a climatic niche shift during biological invasion, and thus support the hypothesis that species can spread into new habitats never been used before by …"
  }, ... other results
  {
    "page_number": 15,
    "position": 10,
    "result_type": null,
    "title": "Forest-attacking invasive species and infant health: evidence from the invasive emerald ash borer",
    "link": "https://www.sciencedirect.com/science/article/pii/S0921800918300314",
    "result_id": "yQkNkX9CTXsJ",
    "publication_info_summary": "BA Jones - Ecological Economics, 2018 - Elsevier",
    "snippet": "… This research could also be admissible evidence into benefit-cost analyses of invasive species management decisions, where the indirect health costs of degradations to environmental …"
  }
]

Disclaimer, I work for SerpApi.