Specifically for a U.S. bachelor's degree, four years is a learning requirement, not a duration requirement. By that I mean that institutions are constrained, by accreditation, tradition, and sometimes by law or regulation, to require learning (work) that can be competed in four years by a student working at "full load." Where I taught, there was a legal requirement that specified the number of credit hours needed for a degree.
Some, perhaps many, students take longer—often much longer—than four years, but they complete the same number of credit-hours as the student who finishes in 3-1/2 years. Some degrees, e.g. architecture, are five-year programs. Similar reasoning applies to masters' programs.
Note that, in the absence of such a learning requirement, a university could just hand out degrees. We call those "diploma mills."
So, the answer with respect to undergraduate education in the U.S. and often elsewhere, is that the amount of learning (work) is established by tradition and often embodied in law or regulation. Someone who can complete that work in fewer than four years is free to do so; someone who takes longer is similarly free to do so.
Doctoral programs are different; the work is set out, at least in general, and can often be completed in about three years, but the student is time-limited to seven or ten years to be sure actual progress is being made.