(I write from a U.S. perspective.) The stresses of the job of academic mathematician include many that have little to do with mathematics itself, but are due to various commodified forms of mathematics. I'll get back to the stresses of research itself at the end.
For example, there're the issues of getting a job after the PhD, getting a tenure-track job after that, and getting tenure after that. These are not research issues per se. Similarly, "getting grants" certainly helps travel to conferences to see other work and promote one's own (though this may be less critical nowadays with Internet), but/and is sometimes used as an "objective external" evaluation of one's research program, so that getting tenure in upper-tier places may be impossible if one does not have an external grant.
The game of "getting grants" is much as what a skeptic would imagine, apart from various degenerative "streamlined" aspects as time goes on. That is, one must propose a thing that one can approximately do, but it is not considered entirely ok to propose to do things one already secretly knows one can do, etc. And "do" has to mean "in a year or two, at most", or it won't help the final report on the grant, nor on the re-application. Also, nowadays, apparently NSF wants its "panels" to favor certain general directions, certain subjects, etc, rather than attempting to seriously evaluate all proposals, thus "steering" research insofar as funding affects it. In particular, it's harder to get grants if you're not doing an "approved" thing.
"Publication in peer-refereed journals" is somewhat less "coerced" than federal funding, "status" is very important for getting tenure, and journals will reject papers perceived as insufficiently high-status to match them. Status is the chief commodity journals possess, and squandering it would be a mistake. But, of course, not all topics of research have equal status. Trying to establish status for a particular topic, or spend one's personal status-currency to do so, is a game in itself.
The long history of mathematics, and the fact that (mostly) things don't "become wrong" in mathematics, mean that there is a looooong backstory, and much low-hanging fruit is gone. (I can see the substantial changes in the 40+ years I've been observing...) Lots of things have been done, and many of the ideas that pop into the heads of people have popped into others' heads before. Stress: how to think of a new thing, that is worthwhile, and, once observed, other people would wish they'd thought of it themselves... but they hadn't??? Sounds like there'd be a scarcity issue, and, in many ways, there is. People have to find "ecological" niches that will generate sufficient status in dept heads' and deans' worlds to get jobs, get tenure.
And, yes, for many of us there is stress in doing the work itself, but there is perverse pleasure in it, or at least no viable alternative. I might claim that it would be silly to try to make a living as an academic mathematician unless one did care about the details so much that confusion or frustration did provide significant stress. Maybe there are better and worse stresses... But, yes, I'm unhappy and bored if I'm not confused or provoked by some mathematical thing, and if I'm confused or provoked I don't sleep well, etc. Just great. :)
But, seriously, apart from the job/business-aspect stresses (which are often hard to overlook), I would indeed claim that there is genuine stress (mostly of a less venal sort) in doing serious mathematical research. Significant scholarship is a prerequisite for not reinventing wheels, and for not thinking that trivial exercises are "research".
Probably the conclusion is that unless the "positive" piquant stress of the confusion of research is sufficiently fun/gratifying, the ugly/bad stress of the business aspect is possibly expensive. But the job/business stresses are universal, apparently, simply manifest in different ways.