I'm conducting a (short) structured literature review to generate new hypotheses. Given the fact I'm on my own, I thought of writing a little scraper (reduces bias) to automatically download the papers for me based on given keywords and search engines (scopus, google scholar, etc.), combined with some visualisation for quickly judging the results for inclusion/exclusion.

But I can't shake the nagging feeling that this has to be a solved problem. It's not that hard and quite a generic issue. I can't find anything though, am I missing something?


Most journals don't like people "scraping" papers, even if it's only a small-scale effort. Tread very carefully. Aaron Swartz was prosecuted (admittedly he did download over 4 million papers), and later committed suicide. Although a small-scale effort is perhaps unlikely to attract much scrutiny, it's still probably against the ToS, which is why no one has published software to automatically download papers. An exception is the arXiv, where you can indeed download everything, if you pay them for the bandwidth.

There are tools to help generate "networks" of papers - Web of Science, Paperscape (for arXiv papers), and ADS Labs (in case you're an astronomer) are some examples. There might be more field-specific repositories that have similar capabilities. But typically these tools will not just automatically download a whole folder full of PDFs for you.

In my experience (and especially if you're dealing with older, scanned articles), searching by keywords and citation maps only will lead you to miss some useful papers. You do need to actually put in some effort yourself in the search! A sound approach is to start by reading a few key literature reviews in the field. Read the useful-sounding papers those authors cite, as well as all of the highly-cited papers in the network. Rinse and repeat.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.