To overcome the 10,000 record limit, use EDirect. The E-utilities in Depth page, says:
For PubMed, ESearch can only retrieve the first 10,000 records
matching the query. To obtain more than 10,000 PubMed
records, consider using EDirect that contains
additional logic to batch PubMed search results
automatically so that an arbitrary number can be retrieved
By downloading the UNIX based EDirect utilities you can run a command line search similar to the following:
esearch -db pubmed -query "2001/06/01 [dp]" | efetch -format uilist
which will produce many more than 10,000 records. Please note, however, that when I ran this particular command (without having particularly studies the documentation) I received the following error:
Actual PMID count 31242 does not match expected total 32637
I don't imagine that it will take much effort on your part to discover the source of the error and overcome it. The important thing is that the command line utility esearch
, along with the other EDirect utilities, gives you much more flexibility than using the direct HTTP interfact.
An alternative approach, of which I give only a bare-bones outline here, is often referred to loosely as "divide and conquer". Note that this is different from the formal, recursive, divide-and-conquer algorithm common in some areas of computer science.
By divide-and-conquer, I am referring to the idea that if you cannot directly solve the problem you want, then break it down into a series of (preferably mutually exclusive but jointly complete) sub-problems, each of which is solvable. For example ... and forgive me, but I am about to illustrate my point with a division method that will not actually work ... consider the possibility that you wish to download the PubMed identifiers (PMIDs) for documents published on 2022-01-01. Assume also, hypothetically, that there are 34,567 such records. You would know that there were 34,567 records because PuMed would return a result that included the string:
<eSearchResult><Count>34,576</Count><RetMax>20</RetMax>
Your next step would then be to ask yourself what sequence of queries you could submit to PubMed which (a) taken together give you all the records you wanted, but (b) taken individually would each return fewer than 10,000 results. One obvious (but, as I said, unfortunately unworkable) approach would be to submit the following 26 queries:
2001/06/03:2001/06/03 [dp] a* [au]
2001/06/03:2001/06/03 [dp] b* [au]
2001/06/03:2001/06/03 [dp] c* [au]
... and so forth. That is, you'd that search for all records published on 2001/06/03 that also had an author whose name began with 'A', and then 'B', etc.
Unfortunately, although this illustrates the divide-and-conquer idea, it will not work for the reason that PubMed requires truncated names (i.e., those where a wildcard *
is used), to be given with at least four (!) initial letters. However, you might experiment, as I have, with submitting queries that use the record creation date (CRDT) as the way of slicing and dicing the records for a particular publication date. For example, whereas
2001/06/01 [dp]
returns 32,637 results, the diced queries:
2001/06/01 [dp] 2000/01/01:2001/06/01 [CRDT]
followed by:
2001/06/01 [dp] 2001/06/01:2001/06/20 [CRDT]
chop the large, unobtainable, parcel of records into downloadable bundles that contain fewer than 10,000 entries. Clearly, you would need to submit further queries with a CRDT range beyond 2001/06/20.