If this was me, I would ponder on this question for one year, then make a final decision. Perhaps taking a sabbatical could help you know what you want. I knew of a colleague who took a one year unpaid leave to ride the country on her motorcycle. She returned for a year, decided that she really just wanted to retire, and retired officially at the end of that school year.
So, would I be crazy to walk away from a tenured job if I don't need it financially?
No. If this is only about finances, there is no reason to keep a tenured position. The whole purpose of tenure is to provide stability in one's career. If you have obtained that stability via other means, I see no purpose in staying in the position out of some sort of perverse respect for tenure itself. I actually think that the issue of tenured professors not retiring when they should is causing a log jam at my former institution.
I don't enjoy stuff like playing department politics.
Would your retirement lead to department politics? If your department chair is planning on you being around for yet a long while, I would give him or her plenty of advanced notice of your plans. If you abruptly left your position, it could cause some politics to arise. It also could lead to difficulty in the future if you do decide to return to academia.
I made some investments a while ago that turned out well, and they generate enough passive income to support my (modest) lifestyle on their own.
I'm not privy to your finances of course, but it seems that you may be banking on these investments supporting you for 40+ years. That's a long time, I'm not going to lie. Projecting returns on investments 40 years hence is next to impossible. This depends significantly on how risk adverse you are. It also depends a lot on if you have a spouse, kids, pets, a house, a car, health issues, etc.
I also worry that it would be impossible to go back to academia if for some reason I wanted to in the future.
This will first depend quite a bit on what your field is (history versus computer science). If you even spent a few years out of computer science, you could be passed by. History (or the like) is probably not so severe in this regard. If it would require any sort of "work" to stay current in your field, then what's the point of being retired? One other question that you would need to examine is, if the time came, what level of academia you wish to re-enter. Getting back into teaching by joining the staff of a junior college would be much easier than jumping into a tenured position at an R1 research university.