Whenever I've heard math professors talk about job prospects in academia, they indicate that the job market has gotten much tougher over the years. Is it true that it is harder now than it used to be to find a good position? If so, why is that?
The number of tenured/tenure track faculty positions in the US has been on the decline in recent years, partly because of a substantial increase in the amount of teaching done by contingent faculty (adjuncts, full time instructors who aren't on a tenure track, and graduate student TA's.)
The number of PhD graduates in mathematics has been fairly steady at around 1200 per year. Most of these new graduates want to ultimately obtain a tenured faculty position, but there are only about 700-800 such positions available every year.
There's a clear inbalance here between the supply of new PhD's in mathematics and the number of desirable faculty positions that they can chase.
A good source of statistics on this is the annual survey done by the American Mathematical Society.
I agree with the earlier answers that the fundamental issue is an imbalance between the number academic positions and the number of people aiming for an academic career. However, there's some context that can help explain how things ended up this way. Everything I say will refer to mathematics in the U.S.; some parts certainly generalize beyond that, but I can't say how far.
It has never been truly easy to get a faculty position in mathematics, but during the period from 1945 to roughly 1970, it was easier than it ever had been before or probably ever will be again. There were huge increases in the number of college students, which required lots of faculty hiring, and federal funding for science increased dramatically (especially after Sputnik). Not everybody could get a great job, but lots of people got good jobs, and a successful grad student could reasonably expect to get at least an adequate job. This is not to say nobody fell through the cracks, but there was no concept of an oversupply of graduate students relative to positions. Instead, people worried about an undersupply.
One consequence is that the number of mathematics graduate students increased exponentially. Lots of universities created new graduate programs, or substantially scaled up existing programs. At the time this seemed like a great idea, because these programs were able to place their students in decent jobs, but when the market crashed the profusion of graduate programs led to a terrible oversupply. (It's difficult to kill a graduate program that's still attracting students, and student enrollment is far less sensitive to employment prospects than one might guess.) We're still suffering from this problem today, in that we have a stagnant academic job market but a system of graduate programs that evolved to feed a rapidly growing market.
The end of the golden age had several causes: the growth of universities flattened out, the economy weakened, and federal funding decreased. Several academic generations had become accustomed to a strong job market, and nobody knew quite how to react when it ended. Some people hoped it was a temporary aberration, which would soon return to exponential growth. Others recognized that the era of exponential growth was over, but hoped that an upcoming wave of retirements would open lots of positions. The net effect is that there was no coordinated response, and it took ten or twenty years before it became clear how the job market was going to evolve. The academic hiring turmoil in the 1990's basically amounted to the realizations that the system wasn't going to fix itself and that nobody else seemed capable of fixing it.
One consequence is that universities have gradually become more demanding in the experience they require, because they can. I know people who got tenure-track jobs in the 60's with no publications yet and without having been in charge of even a single class, but that's unheard of nowadays. These increasing demands put a lot of pressure on graduate students, who risk screwing up their careers if they aren't consistently and impressively productive.
The rate of increase is relatively slow, not clearly visible on a year to year basis, but it becomes clear on a scale of decades. The way the ratchet works is that once too many students adapt and start to meet or exceed the current requirements, universities can become even more selective and the difficulty goes up a notch. In an ideal world, this pressure would lead to increasingly talented and well-prepared generations of mathematicians. There's some truth to this, but unfortunately it's largely an arms race that punishes bad luck or inconsistency more than it rewards exceptional success.
A related consequence is that the job market has become more segmented, as the optimal preparation for different career tracks has diverged. Grad students sometimes imagine a prestige-based hierarchy of positions, with research universities at the top, teaching-focused universities and liberal arts colleges in the middle, and community colleges at the bottom, and they believe that even if they fail to get a job at one level of the hierarchy, they can surely find one in a lower level. They are then shocked to discover that it doesn't work that way: each career track has demanding expectations that don't overlap very much, and not many people can put together compelling applications of more than one sort.
I don't want to catastrophize here: the arms race in qualifications doesn't seem to be driving the fraction of people who get tenure-track jobs to zero. (See here for plenty of statistics. Long-term employment trends are slow and difficult to discern amidst the noise and economic fluctuations, but there certainly isn't a short-term collapse.) Instead, the primary factor in the worsening job market is the stress and difficulty of meeting the increasing demands.
In Germany, increased funding of universities lead to a rise in non-permanent positions, so that nowadays a lot more postdocs compete for the same number of permanent positions.
Let me add a general comment: The specialty about a career in academics is that it (often) takes so many years until you reach a satisfactory (permanent, well-paid) position. People drop out of the process at all stages, and it is not always the good that stay and the bad that leave. On the one hand, you can easily fool yourself to be good enough because it is hard to compare and hard to get an overview. On the other hand, some good people do not like moving every two years until they find something steady, especially if they have a spouse that also tries to build a career. So the job market size in academics relative to the number of qualified applicants is very hard to estimate.