What the publisher is looking for is a book that will attract many readers. If crowds are already clamoring for a book, then the decision to publish it is easy, but that's uncommon. Instead, the publisher generally has to try to predict whether an audience for the book will materialize when nobody knows they want to read it yet.
The "motivation" the publisher refers to is anything that makes people want to read the book. You need to demonstrate to potential readers that the material is interesting and exciting, that reading the book will be useful, that it connects to a larger scholarly conversation, etc.
Right now, it's clear that you feel the book is valuable and would like many people to read it, but you aren't giving potential readers a lot of motivation. For example, in Section 1.4 of the current draft, you write
Somebody might ask, why to study it? My approach relates to traditional general topology like complex numbers to real numbers theory. Be sure this will find applications.
This book has a deficiency: It does not properly relate my theory with previous research in general topology and does not consider deeper category theory properties. It is however OK for now, as I am going to do this study in later volumes (continuation of this book).
Basically, this amounts to saying you don't yet know how it will be useful and you aren't connecting it to the literature. The lack of applications is a serious disadvantage, but not necessarily decisive by itself (some things can be published in mathematics just because they are beautiful or seem like they should be useful). However, not explaining the relationship with previous research is a major problem. It's nearly impossible to get an academic book published under these circumstances.
I don't mean to be overly discouraging, but my impression is that writing a book on your work is premature. Instead, I'd focus on connecting your work with the rest of the field, with the goal of getting a group of other researchers building on your papers. Once you start to get more citations, you can bring them to the attention of publishers as evidence for a community that values your work and would benefit from a systematic presentation in the form of a book.