Here is my analysis for a theoretical subject like Math: (Also assume I am super-smart which I am not):
I assume the stipend would be the same. So will the facility.
There is a change that a good university has a more accomplished professor. But since this is research, I can always go and meet him. Also, many schools allow having a guide from outside the university.
From my experience, the elite university always has a difficult curriculum along with some really book-smart students. So I would be really tensed to pass the exam and hence my research would suffer (Note here I am making a distinction between book-smart students and research smart students which in my opinion are not always same).
University brand is a definite thing. I don't know how crucial this would be after PhD. I for sure know that for undergrad, getting a high paying job was directly proportional to the brand of the university he/she graduated from.
Less pressure as people would assume I am not smart (because of mid-level university) and hence I could excel without any performance pressure except for the one which is required to keep the stipend going.
I would like to have some feedback from the current grad student regarding the validity of my reasoning. Feel free to add your own explanation as well.
NOTE: Not so good doesn't mean a bad university, it just means a mid-table university and good university means universities like MIT, Caltech etc.