Prestige.
Prestige is basically the currency of the academic world. And in the current, grimmer-than-death job market for physical sciences, the prestige of the institution you got your PhD from is the main factor in your post-PhD success, or absence thereof.
It might sound stupid, unfair and cynical. Because it is. Your success should be proportional to your own value, hard-work and scientific output. But with 200 to 300 graduates applying for every opening position, even in third-class colleges and small firms, the prestige of your last-attended institution is the actual cut-off. I have personally met people sitting in hiring committees who openly admitted that any application from someone who didn't graduate from a top-10 US programme or EU equivalent (Oxford, Cambridge, ETH) will go straight to the bin without further reading. Only after this first filter has been applied will they actually start going through your publications, letters of recommendation and actual personal achievements.
So yes, all in all, working for an awesome group in a second-class university might trump working in a mediocre group from an Ivy League. But nothing trumps working in an awesome group in a top-tier university, like dozens, not to say hundreds of graduate students do every year. Those same students you will be faced against when applying for jobs and post-docs. So in today's gigantic battle royale of a job market, where less than 10% science graduates actually land a tenure-track position, if you don't have the full package, you don't stand a chance. That's just how it is.
Personally, I simply wouldn't do a PhD in physical sciences again. And if I did, I certainly wouldn't even consider doing one out of the top-10 programmes of your field. It simply isn't worth your time in terms of employability.