As others have pointed out, oral exams take a lot of time.
I'm not familiar with the US education system, but I'm in a country where, from the elementary school to the university, verbal tests and oral exams are quite common, and I've administered oral exams for about 20 years (with a duration from 20 min to 1 hour, depending on the class). I'll thus try to outline what factors, apart from tradition, allow to administer oral exams in a manageable way.
The most significant factor is probably the structure of courses and exams. In my country, there are usually several exam sessions in a year and students can take an exam in any one of those sessions. And, frequently, for very small courses, professors allow students to take the exam whenever they wish along the year. This means that if you have a course of, say, 100 students, 50 will probably take the exam at the first session, 30 at the second and 10-15 at the third session and the rest along the year (of course, at each session there's also a bit of backlog).
Second, for many courses, there are both a written and oral tests. Those who don't pass the written test are not admitted to the oral test and fail the exam. This means that of 50 students that try the exam at the first session, maybe only 20-40 pass the exam (in the past, just 10 would not have been uncommon), and this further limits the number of oral exams that you have to deliver each session.
Third, there is a certain freedom on how to administer the exams. Some professors will thus make the oral test optional: students who decide to not take the oral test cannot get a top grade (e.g. they can get a B, or equivalent, at most). And, usually, it is remarked that the oral exam can also worsen the grade. This, again, allows to further reduce the number of oral exams (in my experience, only about 10% of the students try the oral exam if it's optional).
In addition to the above points, there is also the acceptance of the stress of the oral exam and the subjectivity, or perceived fairness, of the evaluation (and a few comments here show that this is a controversial point, probably worth of a different question).
"oral examination" "validity" "reliability"
does not paint a happy picture.