There are two fundamentally opposing ideas here. Yes, it would be nice if everyone could afford, and be admitted to, a top-tier university. But that's simply not how prestige works.
- If you admit lower quality students, the number of truly excellent graduates is diluted and your university will lose prestige.
- The loss of prestige may mean a gradual decrease in the quality of faculty, as the very best are hired by other prestigious universities. You'll certainly need to hire more professors to teach the extra students, and can you be sure that all the new hires will be just as good as those you've already got? There are only so many researchers in the world worthy of Nobel prizes.
An everyman's university will never have the prestige of a university such as Oxford, Caltech, or MIT. The key to the success of these top universities is consistency. Damn near every one of their graduates is a top performer, which is very desirable to employers (and also quite attractive for prospective faculty).
Once you lower the barrier to entry, their perceived quality will head downwards, and the flow-on effects will snowball.
There are many very excellent (and open) universities that are not as prestigious as they perhaps deserve, for example ETH Zurich is a truly excellent university that is open to all Swiss citizens who have passed their high school exams. This Times Higher Education ranking page shows somewhat quantitatively the elitism that is at play.