I hope this is not to much rambling, but there is a concrete question in the end. I really like being a mathematician. I like the constant learning, I like working on problems at research-level and I even like teaching university students. I did my PhD and definitely had fun, but during the time, I have also seen aspects of academia that I do not like, most of which fall under "publish or perish", namely the constant struggle for grants, publications and the all-elusive tenured positions.
I'm obviously not alone in this, but let's assume that I have some money, not enough to be overly rich, but enough to keep a modest standard of living without ever having to work. (The details of this are not part of the question, just assume that salary will not be much of a job-motivation.)
Now this got me thinking. I would never want to "retire" at an early age, but on the hypothetical, what if I do not join the fight for tenure and instead effectively tenure myself?
I know that there are many people without university affiliation, who occasionally publish some papers, but are there any who do this full time? There would be enough time to talk with collaborators, write articles I am actually interested in, visit guest lectures at the local universities and the occasional conference on a holiday. Apart from teaching, I would do what everyone else is doing, just on my own. But would it work the same? A possible doubt is, would many people even be interested in working with me? I'm a reasonably competent mathematician but would probably be ineligible for many sources of funding, couldn't offer a big name or any return visits, and would be less motivated to get joint work published in the best possible journal. To put the question succinct:
Is life as a full-time "independent researcher" feasible?
What are other pros and cons? Are there precedents (apart from bored 18th century noblemen)?