3

I'm working on a project to analyze PubMed abstracts over a given period. I'm using the PubMed API (Entrez) to get PMIDs over a given period. I'm submitting queries looking like this:

2001/06/03:2001/06/03[dp]

This would result in all PMIDs with a publication date of the 3rd of June in 2001.

However, for some days, over 10.000 PMIDs were generated.

What can I do to ensure that I get all PMIDs that were generated in a day?

If I could download them from another source, that would be perfectly fine for my application.

5
  • 1
    Go visit your local research librarian and ask them for advice?
    – Jon Custer
    Nov 29, 2022 at 15:30
  • reddit.com/r/labrats/comments/kem5ga/…
    – Anyon
    Nov 29, 2022 at 15:34
  • 2
    Pay the service provider for access without the limit? Nov 29, 2022 at 17:13
  • There is no way (that I know of) to pay to increase the limit.
    – Haffi112
    Nov 30, 2022 at 10:14
  • I have an idea, but first, could you indicate approximately the proportion of dates you have encountered that actually max-out the 10,000 limit? Dec 3, 2022 at 11:12

2 Answers 2

3

To overcome the 10,000 record limit, use EDirect. The E-utilities in Depth page, says:

For PubMed, ESearch can only retrieve the first 10,000 records matching the query. To obtain more than 10,000 PubMed records, consider using EDirect that contains additional logic to batch PubMed search results automatically so that an arbitrary number can be retrieved

By downloading the UNIX based EDirect utilities you can run a command line search similar to the following:

esearch -db pubmed -query "2001/06/01 [dp]" | efetch -format uilist

which will produce many more than 10,000 records. Please note, however, that when I ran this particular command (without having particularly studies the documentation) I received the following error:

Actual PMID count 31242 does not match expected total 32637

I don't imagine that it will take much effort on your part to discover the source of the error and overcome it. The important thing is that the command line utility esearch, along with the other EDirect utilities, gives you much more flexibility than using the direct HTTP interfact.

An alternative approach, of which I give only a bare-bones outline here, is often referred to loosely as "divide and conquer". Note that this is different from the formal, recursive, divide-and-conquer algorithm common in some areas of computer science.

By divide-and-conquer, I am referring to the idea that if you cannot directly solve the problem you want, then break it down into a series of (preferably mutually exclusive but jointly complete) sub-problems, each of which is solvable. For example ... and forgive me, but I am about to illustrate my point with a division method that will not actually work ... consider the possibility that you wish to download the PubMed identifiers (PMIDs) for documents published on 2022-01-01. Assume also, hypothetically, that there are 34,567 such records. You would know that there were 34,567 records because PuMed would return a result that included the string:

<eSearchResult><Count>34,576</Count><RetMax>20</RetMax>

Your next step would then be to ask yourself what sequence of queries you could submit to PubMed which (a) taken together give you all the records you wanted, but (b) taken individually would each return fewer than 10,000 results. One obvious (but, as I said, unfortunately unworkable) approach would be to submit the following 26 queries:

2001/06/03:2001/06/03 [dp] a* [au]

2001/06/03:2001/06/03 [dp] b* [au]

2001/06/03:2001/06/03 [dp] c* [au]

... and so forth. That is, you'd that search for all records published on 2001/06/03 that also had an author whose name began with 'A', and then 'B', etc.

Unfortunately, although this illustrates the divide-and-conquer idea, it will not work for the reason that PubMed requires truncated names (i.e., those where a wildcard * is used), to be given with at least four (!) initial letters. However, you might experiment, as I have, with submitting queries that use the record creation date (CRDT) as the way of slicing and dicing the records for a particular publication date. For example, whereas

2001/06/01 [dp]

returns 32,637 results, the diced queries:

2001/06/01 [dp] 2000/01/01:2001/06/01 [CRDT]

followed by:

2001/06/01 [dp] 2001/06/01:2001/06/20 [CRDT]

chop the large, unobtainable, parcel of records into downloadable bundles that contain fewer than 10,000 entries. Clearly, you would need to submit further queries with a CRDT range beyond 2001/06/20.

2
  • 1
    Thanks a lot! That's a beneficial answer. I have downloaded the annual baseline and used that, but your method is much better aligned with my use case. Regarding the divide-and-conquer, I had tried several combinations and was getting a bit frustrated with the lack of ability to break the search up. For example, the limitations of the wildcard seem pretty arbitrary. But esearch seems to be the way to go then :)
    – Haffi112
    Dec 18, 2022 at 10:44
  • 1
    Note my remark about the discrepancy between two sets of results. It appears to me that eResearch itself uses a slice-and-dice-BY-TIME approach. I have a suspicion that the reason for the discrepancy between some eResearch counts and those from the PMID searches is because eResearch might not slice-and-dice far enough away from the publication data. No doubt a little experimentation on your part will reveal the truth ... and you might even ask the folks as the National Library of Medicine who actually run PubMed. Dec 18, 2022 at 11:18
1

From their help pages one finds:

Download PubMed data

Once a year, NLM releases a complete (baseline) set of PubMed citation records in XML format for download from our FTP servers. Incremental update files are released daily and include new, revised, and deleted citations. The PubMed DTD states any changes to the structure and allowed elements from year to year.

So it seems you should be using their FTP server.

1
  • That is a valid solution, but it requires me to download many gigabytes of data when I am only interested in the PMIDs and abstracts. I would be happy if I could just download the PMIDs as a list with a date of publication attached.
    – Haffi112
    Nov 30, 2022 at 10:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .