I disagree and consider my work fully motivated.
This is a key point. The issue isn't whether there exists compelling motivation, and you may be right that there is. Rather, the issue is whether you have successfully communicated this motivation to readers who have spent far less time thinking about the topic than you have and who lack the perspective that comes from developing a subject from scratch. You'll have to work extra hard to articulate the motivation and communicate it successfully. If the readers still don't understand why are you doing something, then that itself amounts to a communication failure (regardless of whether you feel they ought to have understood). You don't need to communicate successfully with everyone, but you need to do so with a large enough audience.
I'd strongly recommend trying harder to address this before publishing the book. You can consider the editors who have looked over your submissions as a random sample of readers. Many of them fail to understand the motivation behind your work, which suggests that many readers would find it unmotivated even if someone agreed to publish it. Ultimately, the goal should be not just to get the book published, but to get it read and understood. The feedback from publishers is helping you identify what you need to work on to attract readers.
Or maybe should I stop any publishing attempts until I finish writing the second edition?
Are you proposing to publish the first edition while you are already working on a second edition, or to submit the first edition as a draft while telling the publisher that you are working on serious revisions? The latter could make sense, while publishers are unlikely to agree to the former. (Publishing a book takes time, effort and money, and nobody wants to commit these resources to a mathematics book that will quickly be out of date.)
But the fact that you already have large-scale changes in mind suggests that the book might not be ready to submit for publication. You don't need to have completed all possible revisions before approaching publishers, but your chances of acceptance go down if you aren't presenting approximately your best work.
Should I for now remove the last chapter and leave this work to the stage when I will write the second edition of the first volume?
What's the alternative to removing it? If you have in mind fixing the mistakes now, then I can't advise you on which approach is better (since I haven't read the book). If you mean keeping the incorrect chapter in the manuscript and not dealing with the mistakes until later, then that sounds like a bad idea.
I'm afraid that if I happen to die, my work may be lost unpublished.
I can understand that this is a depressing possibility, but I wouldn't worry about it now (assuming you are no likelier to die soon than other people your age). If it makes you feel safer, you could always ask a friend or family member to try to get your work archived somewhere if you died unexpectedly. I can't say they would necessarily succeed, but it couldn't hurt to try.
By the way, the publishers who suggest publishing research papers first have an important point. The mathematics publishing system is set up to work that way, and it is not particularly well adapted to publishing large chunks of unfamiliar research in book form. It's certainly possible to publish research monographs, but you may be making things unnecessarily hard for yourself. Unless you have a very strong reason to prefer a book, I'd recommend trying articles instead.
