I received a master's in math a long time ago and now I am thinking I would like to get a Ph.D. Since I have not been in school, would it be ok to approach a professor I would like to work with by proving a small thing from one of his papers he says is "work to be done" and emailing to him? I already have a proof. Would this add value for him or would he be annoyed? Many people here say you need letters from professors but I have no way for that. This is in America.
In most math PhD programs in the US [I believe most turns to "all" as you approach the pure side of the mathematical spectrum] students are admitted and funded by an admissions committee representing the department as a whole, not primarily by the recommendations (or funding) of individual faculty members.
So if you are contemplating showing this work to a professor in the hope of getting admitted to that department's PhD program: that is not a very direct route. If you are applying to that department anyway, then by all means contact all faculty members with whom you feel a connection and especially in the situation you described. You should also mention this work in your application, e.g. in your statement of purpose. That way it is likely that the admissions committee will contact the faculty member and ask about your case. If you have correctly answered a question from one of their papers [and especially if this answer has not already appeared elsewhere in the literature; it would be a good idea to try to find that out], then they are certainly likely to put in a good word for your application. However, you should understand that students are being judged mostly on the quality of their coursework, the present state of their mathematical knowledge and their potential for future achievements, and not very much on their own research achievements [e.g. because it is common for top candidates to have no research achievements]. So you should not expect this to offset any major weaknesses in your application.
By the way, having no letters of recommendation is a major weakness: in fact, it will cause your application not to be considered at most programs I know. Maybe your idea is rather to get into a correspondence with this professor that will allow them (in time) to write a letter for you? If so: yes, that could work.
By the way, I am currently the Graduate Coordinator [and ex officio chair of the graduate admissions committee] of the mathematics department at the University of Georgia.
If he actually says that it is "work to be done", and you really have answered the question he intended to pose, he would probably be interested to hear from you.
And even if he thinks it's too small a contribution, or if you've answered a question that is different from the one he asked, you have nothing to lose by writing to him.
If the proof is correct, non-trivial, and it was not kind of handwaving "oh, you could do this too", but rather "it would be curious to do, but we have not yet" -- yes! Even if you do not have a second thought of getting a position.
It is always fulfilling to hear about someone picking up your old work. It might even give you a paper together with the Prof. But! Getting a PhD position is a bit uncoupled from fruitful cooperation, as already detailed above.
Is something like an "external" PhD position viable in your country? Basically work elsewhere (possibly part-time) and get the PhD (e.g. from that Prof.) in your free time. I don't know your circumstances, but if you already have a steady job, leaving if for a (typically underpaid) PhD position might be unfulfilling. "External" PhD is about getting the title without working at the university.