How to automatically extract submitted/accepted dates of many journal papers?

Question

I am doing a research of the processing times of papers published in journals in my field. I have noticed that the metrics that the journals advertise (e.g. the Elsevier journal insights) do not correspond to my experience, nor to the recently published papers, so I wanted to make my own survey. (My guess is that they take into account papers which are immediately rejected by the editor without being sent to a review, so the average looks quite favourable. I am more interested in the average time of the papers which are actually accepted.)

I plan to cover all recently (last 12 months) published papers in 10-20 journals of different publishers (e.g. Elsevier, T&F, Wiley), which will result in hundreds of papers. Basically, I will take the date when the paper was submitted, accepted, and published online, and calculate the average per journal.

Is there a way to automatically extract this information?

Answer 1

Have you checked this data is actually made available for your preferred journals? IME not all make their accepted/submitted/first-online dates very easily accessible, though it has improved a bit recently.

If it's there, your best bet is probably to screenscrape the HTML. Some journals provide nice clean XML to play with, but this is usually new online-only titles rather than legacy ones from traditional publishers.

Elsevier use a simple HTML tag (class="articleDates") which contains the core dates -

Received 23 March 2015, Revised 15 May 2015, Accepted 18 May 2015, Available online 9 June 2015

Taylor & Francis have similar information to Elsevier: the element you'd need is again "articleDates", but it unfortunately has a lot of linebreaks in it for no good reason!

Finally, Wiley don't seem to expose submitted/accepted dates (at least not for all journals); "publicationHistoryDetails" just gives first-online, which isn't much help.

Answer 2

This is an interesting question (+1). And I like @Andrew's answer (+1). However, I would like to suggest an approach, somewhat alternative to Web scraping. I mean using meta-repositories and their APIs. For example, you can consider using CrossRef, which offers, along with other services, CrossRef Metadata Services. There is a free of charge subset of this offering, which can be used via what is referred to as End-User Lookup Affiliate (other metadata services seem to be paid).

With that repositories/APIs approach in mind, if you use R programming environment, there is an interesting initiative rOpenSci, which is comprised of an open science-focused set of projects, developing R packages for interacting with various repositories, including meta-repositories. In particular, rmetadata package seems like the project that is the most relevant on the topic (note that it is not a mature project yet). A more mature, but still relevant project is rcrossref package. Hopefully, some other rOpenSci packages also might be of your present and future interests.

Answer 3

Here is a program I wrote to webscrape from a Springer journal. It uses the "Download CSV" link that appears next to the RSS feed icon when listing all articles in the journal Environmental Monitoring and Assessment, URL: https://www.springer.com/journal/10661

To obtain the URL to search on:

• Click on the link to the journal, scroll down past the recent articles, and click on the [ View all articles > ] button.

• To narrow the search, use the Date Published link in the header above the article list.

• There is an RSS feed icon there which is easy to find. However, do not use that icon, but instead use the down arrow icon next to it, which is Download CSV. But don't click on it. Instead, right click on it and select "Copy link address."

• In the code below, in the line which contains os.system, delete the URL after curl and paste the URL just copied above.

• Run the program. You'll end up with an allpapers.csv file containing all the Received, Accepted, and Published dates for all articles matching the publication date range selected.

import os,sys
from dateutil.parser import parse

import requests
from bs4 import BeautifulSoup
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

os.system("curl 'https://link.springer.com/search/csv?date-facet-mode=between&search-within=Journal&facet-start-year=2021&previous-start-year=1981&previous-end-year=2021&facet-journal-id=10661&facet-end-year=2021&query=' > searchresults.csv")

sresults = pd.read_csv("searchresults.csv")
timedicts = []
for URL in sresults['URL']:
    print(f"===== {URL} =====")
    page = requests.get(URL)

    soup = BeautifulSoup(page.content, 'html.parser')

    times = soup.find_all('time')
    timedict = {'URL':URL}
    for time in times:
        label=str(time.find_parent().find_parent().contents[0])
        datetime = str(time.attrs['datetime'])
        if label[0] != '<':
            print(f"{label} : {datetime}")
            timedict[label]=datetime
    timedicts.append(timedict)

allpapers = pd.DataFrame(timedicts)

allpapers.to_csv("allpapers.csv")