I certainly can't and won't try to justify all of the practices of publishers but it is useful to consider a couple of things.
First is that the "market" for any given math paper, say, is extremely small. If the publication is on paper, then the costs, if we consider everything are very high. Even if papers aren't reviewed, they often need to be edited by a professional as not every author is native in the language of the publication. There may be, even in this age, typesetting involved. Plus printing, shipping, etc. There isn't a business model there, so no one would agree to take on publishing of your paper if all costs had to be paid. If they did, the price would have to be extremely high, reducing the market further. As a result, publishers try to avoid a number of costs including payment to authors and reviewers.
In the olden days, authors published their own "tracts" by paying a printer to do the physical stuff and not using anything but (at most) informal reviewing. They would then sell the publications themselves, sometimes on street corners, or contract with other people who did. This was fine if you were in a university town, but not so fine if you wanted a publication created a few hundred miles away. But publishers need a business model, even if it is flawed.
But also, consider a world in which everyone was properly compensated. What would be the characteristics of that world? There are several options. One is that only the most popular writers could get published as that might increase sales to the journal. Competition might ensue, even extreme competition. There would be competition by journals to publish the most popular authors, but there would also be competition between authors to be the most popular authors. It would be hard for a new academic to break in to such a world, perhaps.
Another possible publishing option would be the advertising model. But who would want to advertise in a journal containing abstract, possibly obscure, mathematics. Actually we do have such journals in some fields. They come and go, but they seldom pay authors in any case. And they seldom last very long (a few years at most) making old material difficult to find.
But, the fundamental reason that publishers can use a flawed model in the first place is that, in academia, money isn't the coin of the realm. Recognition by peers is what makes you "rich" in academe. Publishers do provide that - even for new members of the profession. But without a business model they can't and won't.
Let me add a note about another kind of publishing. Popular music. Since it is popular it has a large market and generates a lot of revenue. The artists do get paid. But it is a sad fact of life that very few musical artists, even those with platinum albums, actually make enough from their published music to live on the proceeds. Most of them make a living by going on the road and giving concerts. It never ends. That is why you still see rockers in their 70's giving concerts. Lyle Lovett has gone on record that he has never gotten a statement from his music publisher on which the bottom line was positive. The costs eat up everything - but that includes payments to stockholders, of course, without whom there would be no publishing business at all.
Another publishing model with flaws is textbook publishing. Textbooks now cost around $100US or even more. Of this, the author normally gets around $5. But the costs are high and the market is small. In the case of books, I've always had a professional editor, even for my native language. Figures in the book may need to be remade by a professional for use, depending on the production process. Except for a few courses like Calculus 101 or Economics 101, the market is very small. Very few authors get rich writing textbooks - though many dream to do so.
Let me add that in my lifetime, but long ago, there was a viable not-for-profit publishing model in Mathematics. The American Mathematical Society (AMS) is a professional membership organization. It publishes a number of journals, including the Transactions of the AMS. It is intended for longer papers, not suitable for shorter papers. Anyone could submit a paper to the journal and it would get both editorial and review services, along with, if accepted, production services. This was before the internet and so paper publishing was the only option.
If a paper was accepted there were page charges charged to the paper. They were quite steep, actually. If the author was grant funded then the grant source, normally the government, paid the charges. If the author had no grant, then the charges were sent to the author's institution instead. If, for any reason, the institution refused to pay the charges, they were just absorbed by the AMS and not charged to the author. Thus, the membership as a whole, paid, or page charges were high enough to cover the cases in which they wouldn't be paid. This was a form of socialism, of course, and some people hate that now.
But, for some reason, that isn't the dominant publishing model anymore. I suspect, but don't have the data, that it was simply a case that the model didn't scale well as there was an increase in the number of mathematicians and hence papers.
Non-profit, low cost models do exist, but, as I've said here and in the comments, all of the forces need to be considered in the design of such a model. If you ignore any of them, the model simply won't work for long and its defects will overcome it. But even a failed model, such as the current one, which as failed because it hasn't kept up with electronic communication, has inertia behind it that also needs to be overcome if you want to replace it with a better model.