I am a Ph.D student in Mathematics, and during my undergraduate career, I worked with a professor on a certain topic (topic X, if you will) that strongly interests me. I have now moved to a different institution for my Ph.D studies. This summer I collaborated with him on a paper in topic X, although the collaboration seems to me mostly one sided (as in, I did virtually all the work, and am even unsure as to whether it is ethical for him to be an author of the paper).
I understand why the professor could not prioritize our work because he is really, really busy with other projects in many topics and with his own advisees. Nevertheless, I feel that I am at a point where I could start working independently on questions in topic X that are very interesting to me, and I think his supervision is no longer necessary. Therefore, I'd like to begin working on these questions by myself, or perhaps in collaboration with other people in the area.
I also have a very friendly, advisor-student (though he's no longer my advisor) relationship with this professor that I would not like to lose. Would it be, in some sense, a betrayal to his interests if I start working independently on questions that he is interested in solving too at some point? He is usually very "territorial" about his research projects, and I'm not sure whether he would feel offended that I want to work on these questions by myself, because I think he expects that it would be a collaboration.