Why do some people get STEM PhDs, want a professorship, and end up not getting it?
Is it only about opportunity and competition, or are there PhD candidates that are just not qualified to be professors (why?)?
[Thanks for the clarification, it turns out to be an easy question]
What the reasons are that some people get STEM PhDs, want a research job in academia, but end up not getting it.
Because the ratio between the number of candidates and the number of available jobs in academia makes it mathematically impossible for every candidate to get a job.
Many candidates don't even lack anything to be a good academic, it's just a very competitive job market.
[Edit to answer comments (and the last version of the question)]
are there PhD candidates that are just not qualified to be professors (why?)?
Of course there are:
It's also worth noticing that very few people become professors just after the PhD, postdocs are part of the career process because that's where one can demonstrate the other skills I just mentioned. But the same applies to postdocs: even somebody who has all the skills might not be good enough, it's a tough competition.
* "N" varies depending on many factors: discipline, country, etc.
For the united states, the number of PhD graduates per year is steadily increasing for many years.
However, the federal budget for funding is stagnating and even decreasing:
This does not have to be the situation in other countries:
To better estimate your chances of getting tenure, I would suggest taking a look at the demographic trend in a specific country and how the number of PhD graduates develops.
So the road after PhD towards professorship can be very different in, e.g., Germany vs. US. A rough estimate is always to take a look how many PhD's are graduating during lifetime of a professorship, average over faculty, universities... For physics this will be around ~20-40 in Germany, so your chances are 1/(20-40). In mathematics your chances are much better. But even solid state physics vs. astrophysics this number can vary strongly.
It's basic supply and demand. Too many Ph.D.'s running after too few professorships.
You also need to keep in mind that many grants and PIs have incentives to hire lots of grad students (as cheap, smart, lab techs). That the kids don't have good career prospects is not the concern of the PIs or grants. (Sure, everyone prefers if their people get nice placements. But they aren't going to cut their group size. And the main criteria for taking a grad student is what he will do as a grad student, not his employment prospects.)
That a STEM grad student finds this hard to comprehend (even to anticipate before starting a Ph.D.) baffles me. If someone lacks the curiosity and analytical ability to figure this basic phenomenon out for themselves, how can we be confident they will be shrewd, curious, independent, and creative researchers to recognize new physical phenomena?
I want to add that many PhDs do not want to become professors. A lot of people do a PhD as the final part of their studies without the intention to work as a professor (or see during the PhD that they do not want to stay in Academia). Some do the PhD studies because of the belief that the title helps in their career.
There are far, far more PhD holders than academic positions. In my field, pure math, just about everyone in it is there because they want to be a math professor. (Even if you want the payout of, say, quantative finance, it's probably better to go into physics.) Every point of the process involves a lot of factors outside your control and sheer random chance: how useful your adviser is; whether the research you started on pans out (you only have a few years to produce substantial results, and yet you have to aim high enough for those results to be useful to you); how well you manage to network; your own physical and mental and financial security; and so forth. Just wanting it isn't enough, and there's no way to conjure up a professorship just by putting in enough effort, even if you're quite talented and have a PhD from a prestigious grad school.
There's an anecdote in one of Paul Halmos' books (I think) where he describes talking with his advisor around the time he was leaving grad school, saying he wanted to continue in academia, and his advisor making a phone call to summarily arrange a position for him. That's not a thing anymore. Being good in your field isn't enough; you have to be good at the business of academia, and you have to be extraordinarily lucky.
The requirements and skills needed to get a PhD are different than the requirements and skills needed to get a job offer.
I'll point out that some PhD holders don't hold full-time academic jobs because they don't want the kind of jobs that are available, even though they in principle would like to stay in the academy. This is not quite the same thing as saying there aren't enough jobs (though I don't disagree with that).
For example, in the United States, while in English literature it is now not unusual (I don't know how common) for someone with a terminal doctorate to have a full-time tenure-track-like position at a community college, most scientists would not consider such a position. Similarly, many graduates from more elite programs would rather change careers than assume the responsibilities of a "heavy" teaching load. Some people are very happy to move to another country if they are more interested in a research career, but perhaps many will not be.
My point is that although supply/demand is the overall picture, the OP's question sort of implies there is a monolithic "professor" position, while the reality can be quite different. So perhaps some want a professorship, might even get a job offer, but don't want that one bad enough to take it.
(Personal aside: I'm not sure that this assessment reflects very well on the motivations of people to get a PhD in STEM. If you compare the teaching load of few generations back in mathematics (my field) it looks quite different even at PhD-granting institutions - this paper has some interesting details.)