Starting from a URL, how can I programmatically look up the DOI of an article without scraping the publisher's website?

Question

I have assembled a large collection of URLs pertaining to scientific papers from social media (e.g. "https://www.jni-journal.com/article/S0165-5728(21)00282-4/fulltext)".

I would now like to perform a bibliometric/scientometric analysis of these articles by looking them up on Web of Science, Scopus, or similar. However, most of these scientometric databases use the article's DOI as the entry point for querying. My question is therefore: how can I "convert" a URL into its corresponding DOI programmatically, and without having to scrape the DOI from the publisher's website? Essentially, I am looking for the type of link resolution that Crossref offers, but in the opposite direction (from URL to DOI rather than the other way around).

Any suggestions on how to approach this would be most welcome.

Answer 1

Most likely, you cannot use one approach across all websites. However, most publishers use the same formula for their URLs, so you might be able to do each publisher by itself.

Answer 2

I suspect that you know the answer: There is a database (at doi.org) that provides the translation DOI -> URL. You want the opposite direction, but unless (i) you have access to the database itself, or (ii) the organization that runs doi.org provides an API for the reverse lookup URL -> DOI, you are out of luck: You simply cannot get the reverse lookup that uses the normative data. The best you can do is scrape websites, but because they only provide you with a publisher's view of the data (rather than the normative mapping from doi.org), you may get incomplet or incorrekt data.