Are there universities that cater specifically to shy or reserved people? It seems like many universities have the usual social hierarchy set up of 'popular' student , 'overachievers' , 'sports-inclined' , 'outcasts' or 'rebels' and the very reserved who might also fit into 'outcasts'. Unfortunately nothing seems to be done about this academic social-caste system which is in regular schools and university. Though I'm primarily concerned about the socially reserved students. As young minds are very vulnerable to emotional instability it would be great if reserved students had a special university or at least a special part of a university where they could learn freely without worrying about the problems of socializing. Are there such universities that can be a haven for reserved students?
On academia.SE, undergraduate studies are out of scope, hence, I will answer for graduate programmes here.
For these (master and PhD programmes, and beyond) I am simply not buying into your premise:
usual social hierarchy set up of 'popular' student , 'overachievers' , 'sports-inclined' , 'outcasts' or 'rebels'
Frankly, any graduate programme in the sciences I know of consists primarily of what you would call slightly derogatory "the outcasts" (plus maybe the "overachievers", depending on what you mean with that). Your prototypical high school bully or football jock rarely ends up in a physics PhD programme. In that sense, most graduate programmes are in their own way a haven for such students.
Beyond that, I do not know about any program that specifically caters towards shy people. Further:
where they could learn freely without worrying about the problems of socializing
As long as we are just speaking about "regular" shyness (no clinical condition), I would argue that you would do students a disservice to not "have them worry about socializing" at all. Firstly, even shy students usually enjoy company (if it is the right amount and the right kind of company - I should know, I am also pretty introvert), and, secondly, if you are really painfully shy, learning how to deal with people is arguably a more crucial life lesson than any subject matter you will learn at university. And don't kid yourself - shyness is definitely something that you can learn to work on (again, I should know), given enough practice.
Looking from Europe, and mostly through the lens of podcasts from/about people involved in that, there seems to be much talk about this kind of thing in the U.S. specifically (i.e., "safe spaces" and such). This seems especially prominent in social sciences. So if you really need such a place, then you could look specifically into social sciences at unis and see if they advertise formal "safe spaces". Not all seems to be rosy though, you'd probably do well to research all aspects concerning this; and check if everybody is happy, and if the general mindset is to your liking.
For context, I studied CS in Germany, some 3-4 decades ago. Let me tell you, CS students were not what you call "popular", "sports-inclined" or "rebels", on average.
At no point did I have the feeling that there was an artificial or natural selection working in that direction; nor did I find any kind of "social caste system", hidden or otherwise.
If I really want to come up with some kind of stratification, I would pose that the only real division line I witnessed was between students that were actually interested in the topic and self-driven vs. those that were there because they heard CS was good for earning money, or who were pressured into studying it by their parents (which was unlikely in CS, but did obviously happen in other curriculums, famously business administration). The first kind used to do well without much effort, the second kind not so much.
So another idea would be to look for a field of study that generally attracts more "nerdy" people - i.e., CS, maths, physics or something like that, and stay clear of fields that are maybe not primarily picked by students out of sheer interest, but out of secondary concerns like future job prospects. Obviously the absolute best choice would be a topic that really fascinates you; and then focus on that topic instead of any social strata around you. I find it very very unlikely that you would then suffer mobbing or other social mishandling.
As young minds are very vulnerable to emotional instability it would be great if reserved students had a special university or at least a special part of a university where they could learn freely without worrying about the problems of socializing
This proposition needs some research or reference. First of all, I see no reason that an university, where everybody should be adult already, should be socially worse than the school systems preceding that. At least that was most certainly the case for me - I experienced some mild mobbing at 15-16ish, but that went away in my later highschool years when people slowly became adult and worked out their hormonal issues, and entering uni was immediately just great for a socializing environment. Nobody cared about you whatsoever, and you were free to pick whom to hang out with (or whether to hang out at all).
Secondly, I am not convinced that secluding reserved people to safe spaces does anything to help them become more robust. Again, this probably needs some research...
Thirdly, the time in uni is the final stretch in the life of most people before entering work life, where rules change completely - and any kind of social strengthening absolutely does need to happen before that. So if there were social problems at an uni, the better approach could well be to have mechanisms at hand to handle them, and grow from them, instead of avoiding them in the first place.
The closest thing I can think of, are the Honors Colleges or Honors programs that exist at some universities. Honors classes will tend to be smaller, so even though you may be asked to participate more, you'll have more support and interactions with your professors. Smaller classes may make it easier to find friends as well.
However, these groups look for not just the brightest, but the best. Normally, they will still want well-rounded students--active in their communities or in sports or other activities--not just bright social recluses.
Another solution is to go to a smaller college or university that has a strong program in the field you want to study. Again, smaller classes, closer interactions with professors, etc.
-- edit -- Some schools split incoming classes into small groups during orientation. They will meet, do activities, learn about the college...it's a great place to start finding new acquaintances who may turn out to be friends.
My undergraduate university did this (Trinity U, in Texas). My first friend there was in the that group, but the friend that really stuck was person with whom I shared a music folder in choir.