I am doing a research of the processing times of papers published in journals in my field. I have noticed that the metrics that the journals advertise (e.g. the Elsevier journal insights) do not correspond to my experience, nor to the recently published papers, so I wanted to make my own survey. (My guess is that they take into account papers which are immediately rejected by the editor without being sent to a review, so the average looks quite favourable. I am more interested in the average time of the papers which are actually accepted.)

I plan to cover all recently (last 12 months) published papers in 10-20 journals of different publishers (e.g. Elsevier, T&F, Wiley), which will result in hundreds of papers. Basically, I will take the date when the paper was submitted, accepted, and published online, and calculate the average per journal.

Is there a way to automatically extract this information?

3 Answers 3


Have you checked this data is actually made available for your preferred journals? IME not all make their accepted/submitted/first-online dates very easily accessible, though it has improved a bit recently.

If it's there, your best bet is probably to screenscrape the HTML. Some journals provide nice clean XML to play with, but this is usually new online-only titles rather than legacy ones from traditional publishers.

Elsevier use a simple HTML tag (class="articleDates") which contains the core dates -

Received 23 March 2015, Revised 15 May 2015, Accepted 18 May 2015, Available online 9 June 2015

Taylor & Francis have similar information to Elsevier: the element you'd need is again "articleDates", but it unfortunately has a lot of linebreaks in it for no good reason!

Finally, Wiley don't seem to expose submitted/accepted dates (at least not for all journals); "publicationHistoryDetails" just gives first-online, which isn't much help.

  • An excellent example of what you can do if the data is available, looking at the PLoS journals: metarabbit.wordpress.com/2013/06/05/… Jun 14, 2015 at 19:57
  • Thanks for the suggestion, I didn't notice that it'd be fairly easy to get the info from the HTML. And yes, not all journals (especially Wiley's) provide such data, but luckily they are in a small minority.
    – paginated
    Jun 14, 2015 at 20:38
  • @paginated there's a good chance they'll expose this data soon (the UK HEFCE open-access mandate relies on it, and it would be a lot easier if they reported it, so there's some pressure...) - but for now, I think the html's your best bet. Let us know how you get on! I'd be quite interested to see the results. Jun 14, 2015 at 20:43

This is an interesting question (+1). And I like @Andrew's answer (+1). However, I would like to suggest an approach, somewhat alternative to Web scraping. I mean using meta-repositories and their APIs. For example, you can consider using CrossRef, which offers, along with other services, CrossRef Metadata Services. There is a free of charge subset of this offering, which can be used via what is referred to as End-User Lookup Affiliate (other metadata services seem to be paid).

With that repositories/APIs approach in mind, if you use R programming environment, there is an interesting initiative rOpenSci, which is comprised of an open science-focused set of projects, developing R packages for interacting with various repositories, including meta-repositories. In particular, rmetadata package seems like the project that is the most relevant on the topic (note that it is not a mature project yet). A more mature, but still relevant project is rcrossref package. Hopefully, some other rOpenSci packages also might be of your present and future interests.

  • Great, I was not aware of any of these. Thanks.
    – paginated
    Jun 14, 2015 at 20:38
  • 1
    Anything that doesn't involve screenscraping would certainly be an improvement from a practical perspective. However... it's all a bit moot if the other data repositories don't actually contain the relevant metadata! From a quick check of the Elsevier example it doesn't look like they do... Jun 14, 2015 at 20:41
  • @paginated: My pleasure. Jun 14, 2015 at 20:46
  • @Andrew: I wouldn't make conclusions, based on sample with N=1. Plus, considering current trends, we can expect at least some progress in representativeness and data quality of repositories and meta-repositories in the near future. So, trying this approach can give researchers a sense of comfort (or lack of) for automating bibliographic information discovery and analysis and help build their skills foundation for, hopefully, not-so-distant time, when this approach will become mainstream. Jun 14, 2015 at 20:51

Here is a program I wrote to webscrape from a Springer journal. It uses the "Download CSV" link that appears next to the RSS feed icon when listing all articles in the journal Environmental Monitoring and Assessment, URL: https://www.springer.com/journal/10661

To obtain the URL to search on:

  • Click on the link to the journal, scroll down past the recent articles, and click on the [ View all articles > ] button.

  • To narrow the search, use the Date Published link in the header above the article list.

  • There is an RSS feed icon there which is easy to find. However, do not use that icon, but instead use the down arrow icon next to it, which is Download CSV. But don't click on it. Instead, right click on it and select "Copy link address."

  • In the code below, in the line which contains os.system, delete the URL after curl and paste the URL just copied above.

  • Run the program. You'll end up with an allpapers.csv file containing all the Received, Accepted, and Published dates for all articles matching the publication date range selected.

import os,sys
from dateutil.parser import parse

import requests
from bs4 import BeautifulSoup
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

os.system("curl 'https://link.springer.com/search/csv?date-facet-mode=between&search-within=Journal&facet-start-year=2021&previous-start-year=1981&previous-end-year=2021&facet-journal-id=10661&facet-end-year=2021&query=' > searchresults.csv")

sresults = pd.read_csv("searchresults.csv")
timedicts = []
for URL in sresults['URL']:
    print(f"===== {URL} =====")
    page = requests.get(URL)

    soup = BeautifulSoup(page.content, 'html.parser')

    times = soup.find_all('time')
    timedict = {'URL':URL}
    for time in times:
        datetime = str(time.attrs['datetime'])
        if label[0] != '<':
            print(f"{label} : {datetime}")

allpapers = pd.DataFrame(timedicts)


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .