In the US, in most doctoral programs, the goal is to produce a dissertation, not publications. This was, I'd guess, more true in the past, but remains largely true. A dissertation is usually a relatively long work on a small, tightly focused, area of math. The dissertation might result in one or more publications after finishing the degree but getting involved in the publication process, along with the waits and revisions, isn't usually part of the process prior to graduation.
If you don't produce a dissertation, you don't finish. But most people finish successfully, though some take longer than others.
But, if you don't produce a dissertation it is likely because you didn't listen to anyone while you were a student. Students have advisors and advisors have a responsibility to get their students through the program. They suggest problems or, at least, small areas of math where exploration is likely to prove fruitful. They also give you feedback on what you do and suggest modifications if needed.
If you ignore your advisor's advice it is harder to advance. You aren't locked in a box, all alone, for a few years expected to emerge with some new mathematical field. Results in dissertations can be quite small or more broad. But they are very focused. Advanced graduate study pushes and guides you to that focus.
However, some problems that students attack prove too hard to find a solution in the time available even with the advisor's help. Some students have to "salvage" a dissertation out of failed attempts. But more students, I suspect, are advised to drop a problem if it seems, early on, to not be fruitful. And note that in math, proving negatives can be as valuable as proving positives.
There is a Three Bears issue in math (too hard, too easy, just right). In my own studies I experienced this directly. The first problem I worked on proved too easy and I could produce theorem after theorem daily. There was no substance. The second problem was too hard. After a few weeks it still seemed like a perfect titanium sphere with no possible cracks for entry. Nada. The final problem was "just right" and with work and developing insight a nice dissertation (later published) emerged.
With your advisor's help, choose a tractable problem. Work hard. Succeed. Or, at least, learn that this isn't what you really want.
Note that working mathematicians seldom work in isolation and collaborate widely, sharing ideas. Colleagues substitute for advisors in a sense. Ideas fly around. Some of them result in nice papers. Some are discarded.