Are there universities that cater specifically to shy or reserved people? It seems like many universities have the usual social hierarchy set up of 'popular' student , 'overachievers' , 'sports-inclined' , 'outcasts' or 'rebels' and the very reserved who might also fit into 'outcasts'. Unfortunately nothing seems to be done about this academic social-caste system which is in regular schools and university. Though I'm primarily concerned about the socially reserved students. As young minds are very vulnerable to emotional instability it would be great if reserved students had a special university or at least a special part of a university where they could learn freely without worrying about the problems of socializing. Are there such universities that can be a haven for reserved students?
On academia.SE, undergraduate studies are out of scope, hence, I will answer for graduate programmes here.
For these (master and PhD programmes, and beyond) I am simply not buying into your premise:
usual social hierarchy set up of 'popular' student , 'overachievers' , 'sports-inclined' , 'outcasts' or 'rebels'
Frankly, any graduate programme in the sciences I know of consists primarily of what you would call slightly derogatory "the outcasts" (plus maybe the "overachievers", depending on what you mean with that). Your prototypical high school bully or football jock rarely ends up in a physics PhD programme. In that sense, most graduate programmes are in their own way a haven for such students.
Beyond that, I do not know about any program that specifically caters towards shy people. Further:
where they could learn freely without worrying about the problems of socializing
As long as we are just speaking about "regular" shyness (no clinical condition), I would argue that you would do students a disservice to not "have them worry about socializing" at all. Firstly, even shy students usually enjoy company (if it is the right amount and the right kind of company - I should know, I am also pretty introvert), and, secondly, if you are really painfully shy, learning how to deal with people is arguably a more crucial life lesson than any subject matter you will learn at university. And don't kid yourself - shyness is definitely something that you can learn to work on (again, I should know), given enough practice.
The closest thing I can think of, are the Honors Colleges or Honors programs that exist at some universities. Honors classes will tend to be smaller, so even though you may be asked to participate more, you'll have more support and interactions with your professors. Smaller classes may make it easier to find friends as well.
However, these groups look for not just the brightest, but the best. Normally, they will still want well-rounded students--active in their communities or in sports or other activities--not just bright social recluses.
Another solution is to go to a smaller college or university that has a strong program in the field you want to study. Again, smaller classes, closer interactions with professors, etc.
-- edit -- Some schools split incoming classes into small groups during orientation. They will meet, do activities, learn about the college...it's a great place to start finding new acquaintances who may turn out to be friends.
My undergraduate university did this (Trinity U, in Texas). My first friend there was in the that group, but the friend that really stuck was person with whom I shared a music folder in choir.