I'll speak as a person who has been on graduate admissions committees in a field that ends in "-physics" at the university with a top five ranked graduate program and also at one that is in the top ten, but not top five.
First, context. It is destructive to take on weak students who drain resources and then drop out or worse, stay to the bitter end and do a weak or failed PhD. They drag other students and faculty down and take the place of more worthy applicants. So committees are very hesitant to take risks. The admissions rate at these elite places is like 5-10%, and that's already among a self-selecting group of usually strong applicants.
If you attended a major research institution, the risks are lowered for the committee. If you did your undergrad at MIT and a reference letter mentions you were the top physics student there in five years, you would almost certainly be admitted to an elite graduate program. (An average undergrad physics student from an elite university is not going to get into a top graduate program.)
If you went to a small school, the same statement that you were the top student in the last five years gives no real information. How many physics students graduate in five years? 20 students? What was this physics cohort like, that you were the best among them? What is near-certain is that it wasn't MIT caliber. What was the quality of your education? Did your textbooks end with "...for scientists and engineers" or begin with "Principles of..."? You can understand these sorts of questions.
The old equalizer was the physics GRE, but unfortunately the current trend of ditching the physics GRE as an admission requirement, for "equity" reasons, definitively hurts strong students from smaller institutions, as it makes committees much less willing to take a chance on you. (If the institution accepts physics GRE scores, DEFINITELY submit yours if they are solid, like 800+.)
You will need concrete evidence that you can thrive in graduate school. This includes most of the below:
- refereed publications, especially as a lead author
- poster presentations or talks at major conferences
- solid research talent described in multiple letters of recommendation, especially strong if it comes from a mentor at an R1 who can compare you favorably to other excellent students who went on to elite graduate schools
- Basically perfect grades
- Academic awards, either from your university or better, at a national level
- Funded graduate fellowships (NSF, etc). These make you financially more easy to admit.
If you went to a small liberal arts school or something like it, to have a chance at a tippy-top program--eg, Harvard, Princeton, Caltech, Stanford, Berkeley, etc--you need all these things, and your chances are still not good. For example, my graduate school friend group at one of those places included an international physics olympiad gold medalist, a Rhodes scholar, and someone who got the highest score in all of India on the national college entrance exam. People like that are going to be your competition.
The good news is, you can still do exceptionally well at a say, top fifteen but not top five program, and there, your chances of admission are way, way, higher if you can meet conditions 1-5 above. Good luck!