NIH posts their study section members who review grants online (e.g. Aging Systems and Geriatrics Study Section (ASG)'s roster is provided here). I was trying to search the study sections to find whether a particular reviewer served as a study section member. However, this is not easy, because their website does not provide a search function that does this (as far as I can tell), and the website NIH study section rosters that contain grant reviewer names disallow search engines from crawling and indexing them via robots.txt.
User-agent: * Disallow: /
Clearly, I can do a manual search by going to each page and finding if any of the study section contains the name I'm looking for, or manually make a database for personal use.
What are the possible rationales for disallowing search engine indexing reviewers?