9

For some concrete study, I need to find for some keywords, papers that have just one author. I used PubMed, Scopus and ISI Web of Science, but in any case, I could not find a way to filter bibliographic search results by a specific number of authors, or a number less than some exact number of authors. Anybody knows how to do this with mentioned tools (PubMed, Scopus and ISI Web of Science) or others?

9
  • 5
    For some concrete study --- When I began reading your question I thought it was about a study involving concrete! (e.g. From a materials science perspective, from an engineering construction perspective, from a business cost perspective, etc.) However, as I was reading your 2nd sentence I realized what you meant . . . Commented Mar 23, 2018 at 6:35
  • anyway, it could also be related to scientific studies about concrete, asphalt, etc, why not? and ... is it relevant? Commented Mar 23, 2018 at 9:58
  • 1
    After a quick google search: The author of this paper (arxiv.org/abs/1206.4863) claims that "of the 1.3 million publications [...] 2011 [...] in the Web of Science database, 89% had more than one author.", so it might be worth to ask him how he obtained this number, maybe you can use his method.
    – Dirk
    Commented Mar 26, 2018 at 9:44
  • 1
    yes, that might be the reason. and a good approach. what I will do, is just to download the specific results of my search, as CSV, and then process them in excel for one single author records Commented Mar 27, 2018 at 7:52
  • 2
    @flow have you tried to contact support team of science direct or ISI? they are fast with replies
    – SSimon
    Commented Mar 28, 2018 at 17:10

3 Answers 3

3
+100

PubMed has an interface which you can call from a script. The intend to develop it is exactly your class of problem, which cannot be solved from the provided user interface.

This is the main page of NCBI Entrez API: https://www.ncbi.nlm.nih.gov/books/NBK25501/

What you need to do is to query PubMed by keyword(s), for example this is a search by "concrete":

https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&retmode=json&retmax=100&sort=relevance&term=concrete

Do multiple queries, to cover your field, for example you can also consider "brick" or "cement".

It returns a list of publications. For each publication, you would have to check the number of authors and keep ones with a single author. For each publication, you shall call:

https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=29510510&retmode=json

Determine the size of the vector "authors", and only keep the ones of size one.

3
  • can you elaborate about how to determine the size of vector authors? Commented Mar 31, 2018 at 11:09
  • sorry for downvote, that was completely unintentional. I'll change that as soon as possible!
    – OBu
    Commented Apr 1, 2018 at 15:53
  • @OUb no problem, I've upvoted yours, good work!
    – user83564
    Commented Apr 1, 2018 at 17:11
3

Based on @Razvan P's hint, I wrote a little python3-script which solves your problem:

'''
Created on 01.04.2018

@author: OBu
'''

import requests
import json
from collections import Counter # for histogram

eutils_basepath = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/'
DB = 'pubmed'           # please modify for other databases
RETMAX = '100'          # max 100 results - modify if needed, maximum = 100.000
SEARCHTERM = "concrete" # replace with your search term

# Now build the search URL:
search_url = eutils_basepath + 'esearch.fcgi?db=' + DB + \
                               '&retmode=json&retmax=' + RETMAX + \
                               '&sort=relevance&term=' + SEARCHTERM
# for additional search parameters or mor complex search terms see examples in
# https://www.ncbi.nlm.nih.gov/books/NBK25500/#chapter1.Searching_a_Database
# or the full doc under 
# https://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.ESearch

s = requests.Session()
r = s.get(search_url)
if r.status_code != 200:
    raise ConnectionError("Search failed with error code " + str(r.status_code))
search_results = json.loads(r.text)

#show some statistics
print(f"{search_results['esearchresult']['count']} publications found.")
if RETMAX < search_results['esearchresult']['count']:
    print(f"Warning: Only the first {RETMAX} publications are processed.")

# walk through all rerieved ids and fetch detailed publication information
# An alternative soloution could use one single query based on the previous search results as shown in 
# https://www.ncbi.nlm.nih.gov/books/NBK25500/#_chapter1_Downloading_Document_Summaries_
# This would reduce the server load
histogram = Counter()
for pub_id in search_results['esearchresult']['idlist']:
    #print(f"Fetching {pub_id}", end=" ") # uncomment for a more verbose versione
    # Now build the fetch URL:
    fetch_url = eutils_basepath + 'esummary.fcgi?db=' + DB + '&retmode=json&id=' + pub_id
    r = s.get(fetch_url)
    if r.status_code != 200:
        raise ConnectionError(f"Fetching of publication {pub_id} failed with error code {r.status_code}")
#     else: # uncomment for a more verbose versione
#         print("...success!") # uncomment for a more verbose versione
    fetch_result = json.loads(r.text)
    authors = fetch_result['result'][pub_id]['authors']
    if len(authors) == 1:
        print(f"UID: {pub_id}, author: {authors[0]['name']}, title: {fetch_result['result'][pub_id]['title']}")
    histogram[len(authors)] += 1    

print("Histogram: (number of authors, number of papers with that many authors)")
print(sorted(histogram.items(), key=lambda x: x[0]))

You will need python 3.6 or above to run this script (please remove f-strings for earlier versions, and you'll hvae to install "requests" via pip install requests.

The script searches for the SEARCHTERM in pubmed and for the search term concrete (I like this running gag ;-) ) produces an output like

14125 publications found.
Warning: Only the first 100 publications are processed.
UID: 28844248, author: Baroody AJ, title: The Use of Concrete Experiences in Early Childhood Mathematics Instruction.
UID: 28772472, author: Wang XY, title: Modeling of Hydration, Compressive Strength, and Carbonation of Portland-Limestone Cement (PLC) Concrete.
UID: 29159238, author: Paul SC, title: Data on optimum recycle aggregate content in production of new structural concrete.
UID: 27012788, author: Kovler K, title: The national survey of natural radioactivity in concrete produced in Israel. 
Histogram: (number of authors, number of papers with that many authors)
[(1, 4), (2, 15), (3, 21), (4, 24), (5, 19), (6, 10), (7, 3), (8, 2), (10, 2)]

It should not be too difficult to modify the script for other search tasks...

If there are questions on how to use the script, please ask!

1
  1. Search your query on Pubmed
  2. The Display Options button on search results page has option for showing 200 entries. Combined with Show more results button on page bottom, any number of entries in principle can be displayed on the same page.
  3. Open Console in Developer Tools. (Shortcut Ctrl+Shift+J)
  4. Copy below code and Ctrl+V in Console.

let text = document.getElementsByClassName('docsum-authors full-authors'); let result = []; for(i=0;i<text.length;i++){if(text.item(i).innerText.match(/,/g)==null){result.push('%%%'+(i+1)+'%%%'+text.item(i).innerText)}}

  1. If no warning flashes on pressing enter, the code has probably worked and the single-author articles among all articles currently loaded on the page are saved in result variable.
  2. Type result in next line and press enter.

For shorlisting papers with no more than, say; 6 authors,

let text = document.getElementsByClassName('docsum-authors full-authors'); let result = []; for(i=0;i<text.length;i++){if(text.item(i).innerText.match(/,/g)!=null){if(text.item(i).innerText.match(/,/g).length<6){result.push('%%%'+(i+1)+'%%%'+text.item(i).innerText)}}}

where the same code works when replacing <6 with ==7 for papers having exactly six authors.

Haven't checked with other databases, but provided the format of entries is similar (list of authors separated by commas in an html tag with unique class name) above code should work with replacement of class name ("docsum-authors full-authors" for Pubmed) and maybe the loop range/if conditions.

Deep apologies for the simplistic language, no disrespect intended to anyone on the thread. My only intentions were being equally relevant to a broad audience having no familiarity with HTML or Console.

P.S. the second (longer) code is not a generalization of the first, since it can only shortlist papers with at least two authors. Only the first can be used to get single author papers.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .