Well, it depends how strict you are with the terms "open" and "free":
Most European countries I am familiar with (Austria, Germany, Sweden, Switzerland) are actually fairly open in terms of admission. There are always some basic requirements (e.g., candidates need to have a high school diploma or comparable), and sometimes there are entry examinations, but in general there is no US-style admissions system.
As an example, in Sweden university programs can define their own objective admission criteria (e.g., specific grades in specific high school subjects) and how many students they are willing to admit, but they cannot rank students themselves - they need to accept at least as many eligible students as advertised based on a ranking generated centrally (usually based on high school grades, as far as I know). Students will generally get admitted somewhere, but some of the best schools (e.g., KTH, Chalmers) you will need rather good grades to get in.
In Germany, the system is comparable to Sweden in the sense that your high school grades largely determine whether you can enroll in a specific program at a specific university through a system called Numerus Clausus.
In Austria, historically every eligible prospective student could enroll freely in any program of any university (there was no further admission system, aside from basic requirements such as having a high school degree or equivalent). This is still the case for many programs, but in some high-demand programs (e.g., medicine) they have now started to experiment with entry exams.
These systems are all quite different to US-style admissions systems. There are no application packages, interviews, or recommendation letters, and it's generally not the university that actually makes the decision (instead there is some broken but objective kind of ranking criterium, such as grades or test results). Further, these systems are often more about routing than they are about admission - for instance, in Computer Science in Austria, everybody who wants to study CS can do so in Austria, but not everybody will be able to study at TU Vienna (historically the most popular place to study CS in Austria). It's also important to understand that the difference in quality of the schools is actually not dramatic, so "getting into" TU Vienna is not a huge job market advantage - people mostly prefer studying in some cities over others.
In Switzerland, as far as I know admission is still completely open - every eligible Swiss citizen can choose to enroll at ETH or EPFL (international top universities). The "selection" here mostly happens at the first semesters of study - for ETH, I know that many, many students will give it a shot and drop out within the first weeks, after realizing that they will likely not be able to keep up with the highly intense workload and intellectual challenge.
Education in all these European countries varies between free and (relatively) cheap. Taking Austria as an example again, studying was historically indeed free at all public universities (which were basically all of them - private universities are a new trend around here, and not one that has really gained any traction). Since around 2000, we have been flip-flopping back and forth between "free" and a study fee of around 400 USD per semester (depending on which political party was governing the country). Tuition for non-EU citizens is about twice as high (so about 1600 USD per year of study).
In Sweden (and, I think, in Switzerland and Germany) there is no fee for eligible citizens (EU citizens in the case of Sweden and Germany, Swiss citizens in Switzerland). At least in Sweden and Switzerland, tuition for citizens of other countries is at least in the same ballpark as in the US (still a bit cheaper, but far from "free").
However, I assume that placements are competitive and have strict entrance and performance requirements.
To summarize, this assumption does not really work out. Systems differ, but in general universities are fairly open for eligible students (although entry tests and grade-based systems are increasingly seeing use).
So how do universities cope? Mainly by scaling up. Classrooms in Europe can be large - classes with 500 or 1000 students are not a rare sight in popular programs in Austria, Switzerland, or Germany (but not in Sweden, as programs can limit how many yearly students they accept - often in the range of about 100, give or take). It is often accepted that many students will fail and drop out - when I studied CS in Austria, it was not uncommon to partake in early exams with fail rates considerably above 50%, and about half of my classmates switched programs / universities during the first year of studying.