Rather than a direct answer to the question, the following is more of a frame challenge. I think the assumption in the question about why a mathematical manuscript can be rejected ingores a number of very relevant reasons.
First of all:
The editor might come to the conclusion that the results or techniques in the manuscript are not sufficiently innovative to warrant publication in the journal to which the manuscript was submitted.
On a similar note, the editor might consider the topic of the manuscript to be not sufficiently "interesting" (however the editor interprets this word) to warrant publication in this specific journal.
The above two cases will often result in desk rejection of the manuscript rather than peer review. It might, however, still be likely that the manuscript can be published in a different venue.
If, however, the manuscript makes its way to peer review, then the reviewer(s) will typically be asked to recommend whether the paper should be accepted, or revised, or rejected. The final decision will, of course, still be within the discretion of the editor, but in my experience, editors will often follow this recommendation.
Now, there are many potential reasons why the reviewer(s) might recommend rejection, even if they do not find an unsalvageable mistake in it:
They could find that the results are not sufficiently "substantial", i.e. they might be more or less clear to experts in this particular field.
The results and proofs might be written in a very sloppy way, which makes it extremely difficult to even determine if they are correct.
The writing of the paper might just be outright bad, thus making it a real burden to even read the paper.
It might turn out that the authors are completely unaware of large parts of the relevant literature.
The last three points might, in principle, be fixable, assuming that the authors take them seriously and put sufficient effort into it. However, this does not imply that the reviewers were necessarily obliged to recommend a revision of the paper instead of a rejection in such a case. A good reason to actually suggest rejection in some of these cases if the following experience that I made quite often when got the first requests to review papers:
The manuscript suffered from several of the issues mentioned above; I wrote a very lengthy report where I pointed out in detail all the problems that I encountered and suggested a major revision of the manuscript. The authors apperently took this as an encouragement that there paper were "close to acceptance", and instead of thoroughly improving the manuscript, the tried a "minimally invasive" approach to superficially deal with the issues I had raised, but did not make a real effort to substantially improve the paper - leading thus to yet another long report in the second round of review.
After I had become more experienced and already used to this pattern, I changed my recommendation practice. If a paper exhibits too many issues of the types mentioned above, I will typically point out several of them, include some general recommendation to the authors how they should improve their paper if they plan to submit it elsewhere, and recommend to reject the paper.
In such a case, recommendation for rejection (rather than for a major revision) clearly signals to the authors that, from my perspective, that paper is not close to being publishable - so much more work is needed to improve it than would be the case during a "major revision".
For the majority of papers which I, as a reviewer, recommend to reject, the reason for my recommendation is actually this "not close to being publishable" issue rather than non-fixable mathematical errors.
TL;DR: Apart from mathematical errors that cannot be fixed (or cannot be fixed within a reasonable time frame), there are many more potential reasons why a math paper can be rejected by the editor or recommended for rejection by the reviewer(s).
