Crudely, it seems better to study at a big department - one has more peers, more seminars, and is also more likely to find a faculty member with the same interests if one's interests change during the program. However some top-tier departments appear to prefer to stay small, e.g. Caltech as a whole. That seems to imply there's a downside to becoming a big department, which is part of the reason they haven't expanded (I'm guessing they aren't staying small because they lack the funding - they're top tier after all).
For the student, is it preferable to study at a big or small department? What are the pros and cons of each?