Based on your description, getting a Masters degree is not a good idea. Several reasons:
- You can't be sure your mother will remain healthy.
- You currently have no savings, and you're already 38. There's not much time left to build up a retirement fund - and as you point out, academia does not pay well.
- You already have student loans to pay for.
- Getting a job in academia is hard. "There's a chance I could never achieve tenure" is a very calm way to put things; the real odds of you never achieving tenure is probably in the vicinity of 80% in the humanities.
Let's take the rosiest result that you could possibly get by pursuing an academic career. Your mother remains healthy. You are able to fully fund your studies, without incurring extra debt. Your current debt is paused until you finish studying, after which the interest rate is 4%. You finish in 6 years. After graduating you immediately land a tenured professorship paying $60k a year, from which you are able to save $30k a year. You use the savings to first pay down the debt, then invest all of it into the S&P 500, which returns 10% a year (this itself is risky, since you need to save some money for a rainy day, but we are considering the rosiest possible result).
- Six years from now: you are 44 years old, net worth -$25,000.
- Seven years from now: you are 45 years old, net worth $4,000.
- Twenty two years from now: you are 60 years old, net worth $1,065,200.89.
If we further assume that $1 million is enough to retire on in 22 years' time (itself a shaky assumption), then in the rosiest scenario, you are just barely across the finish line.
You could make the model more complicated of course. Perhaps your $60k/year salary will also increase at the rate of inflation (although if taking inflation into account, your expenses would also increase), perhaps you are willing to delay retirement to 65, etc. But the margin of safety you have is tight, even in the rosiest possible scenario. What if your mother falls sick? What if the stock market crashes right as she does? What if you are involved in a car accident and need surgery? What if you can't live on $30k a year (note you have to pay taxes out of this number; further, even if you can, what about your spouse/children)? What if you can't land a tenured position?
tl; dr: don't do it.
Edit: in the more general case, one would modify the model above to take on more realistic assumptions. For example you could say that immediately after graduating, you do a postdoc that only pays $40k a year for 6 years, during which you save $20k a year, before landing a tenured position. This changes your net worth in 8 years' time to $13,760, your net worth in 12 years' time to $122,248.02, and your net worth in 22 years' time to $843,014.89.
Perhaps you decide that when you become a professor, you'll buy your own house for $100k. This further modifies the calculation above to make your net worth in 12 years' time $22,248.02, and your net worth in 22 years' time will be $583,640.65 (+ however much the house is worth).
If you then decide that you will work longer before retiring, your net worth in 27 years' time will be $1,141,427.40. In fact, if you play around with the calculator, you'll find that time really is money. 10% more time beats earning 10% more salary! That's why 23-year olds can follow their hearts while older people must be more conservative if they want to retire comfortably.
You ask further about how to model probability of success in becoming a professor. To do this, search up an alternative career option (preferably several) for people with your intended degree. Next, use your local jobs portal to look for jobs for people with that degree. Finally, check salary websites for how much these jobs earn. Now instead of the $60k/year salary assumed in the calculations above, you put in whatever your new salary is, and rerun the calculations. Remember there's an ~80% chance you never become a tenured professor, so a plan B is important.
If after running all the calculations with realistic assumptions, you find that you can reach the finish line with room to spare (how much margin of safety depends on your risk appetite, but you can always make more pessimistic assumptions and see if you still get there), then a PhD is financially feasible.