NIH posts their study section members who review grants online (e.g. Aging Systems and Geriatrics Study Section (ASG)'s roster is provided here). I was trying to search the study sections to find whether a particular reviewer served as a study section member. However, this is not easy, because their website does not provide a search function that does this (as far as I can tell), and the website NIH study section rosters that contain grant reviewer names disallow search engines from crawling and indexing them via robots.txt.


User-agent: * 
Disallow: /

Clearly, I can do a manual search by going to each page and finding if any of the study section contains the name I'm looking for, or manually make a database for personal use.

What are the possible rationales for disallowing search engine indexing reviewers?

  • 4
    I'm voting to close this question as off-topic because it's an administrative question for the administrators of a particular website. – EnergyNumbers Sep 13 '16 at 18:28

@Memming, this could be done for any number of reasons. The two I can think of off the top of my head is 1) to prevent site slowdowns especially due to what is likely to be the duplication of information developed by indexing the NIH site and 2) to protect (to some extent) the privacy of the participants.

One suggestion, have you tried using Google to search the site? Search is not the same as indexing. So if you're looking for sections with "Joe Brown" you can try searching "Joe Brown site:csr.nih.gov". I was able to get results with this approach using a name I picked at random from the study sessions.

  • Thanks for the reply! Interesting that you got results using the "site:" restriction with Google. I do not get any results from the "intranet.csr.nih.gov", only "public.csr.nih.gov". – Memming Sep 13 '16 at 18:42
  • 1
    Notice that I dropped off the first part. Just csr.nih.gov. – Dave Kanter Sep 13 '16 at 18:43
  • I did try that first. It only gave me results from public.csr.nih.gov. Google search is known to give heterogenous results, so this might be one such case. – Memming Sep 13 '16 at 18:49
  • Yeah, hacking Google is a clumsy and inexact science but sometimes it works... ;-) – Dave Kanter Sep 16 '16 at 16:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.