Say, why is the book How The Immunse System Works (by Lauren Sompayrac) regarded as a "textbook" but, say, The Better Angels of Our Nature is not? There are thousands of non-fiction books, some of higher quality than others, some more technical or more difficult than others, some much bigger than the average (like Russell's History of Western Philosophy), but only a small fraction of them are called "textbooks". What's the difference? On what basis is a book judged as a "textbook"? Size? Academic rigor? Amount of detail? Level of technicality?
Technically, if there is a course in which it can be useful, then it is a textbook. So To Kill A Mockingbird fits the bill and it isn't even non-fiction. But I guess that isn't the answer you are after.
I think a big part of it is just a marketing decision by a publisher. If there are enough courses to which a book might apply it may be sold to a captive audience (students) as a textbook rather than to a wider audience but with less compulsion to buy.
Textbooks, however, tend to have student exercises in them to reinforce the work. The idea is that "learning is required". Non fiction usually doesn't have exercises and learning is optional. If a publisher decides to publish a work as a textbook there will be pressure on the author(s) to provide exercises and other instructional materials. There will also be pressure to provide, say, new editions every other year.
Textbooks also tend to be directed to a certain level of student attainment. Some books are written specifically for beginners. But at advanced levels, there is probably less distinction, as students then have the background needed without the pedagogical aids of lower level texts (we hope).
Often, but not always, textbooks are published to very high manufacturing standards with non-acid paper, sewn signatures, and such, imagining that a student will want to keep such books for several years. This is normally less of a concern in the popular press. I've had a number of books for over fifty years. (This attention to quality may be less true than it once was, I suppose.)
I think you can find a wide range of quality in either category, as well as a fairly wide range of difficulty. Nor is size an indicator. Most elementary Calculus books are fairly thick, and intended for two or three terms of study.
Cost is another factor. Text books are expensive since the intended audience is often quite small and balkanized into several different books. Non fiction, if well written, appeals to a wider audience and so can be sold for less, making the profit from volume.
I don't see a lot of distinction based on your other suggested criteria. Either can be detailed or not, more or less technical (based on "level"), etc. But textbooks are designed to be learned from whereas general non-fiction, like fiction, is designed to be enjoyed. There are a few truly beautiful textbooks, of course.
For "fun" I read a fair number of archaeology and anthropology books. None of them are intended as textbooks, but most would serve as such, since they have extensive bibliographies aiding a student in research. They are probably less useful, but not useless, for beginning students, but provide a good place to start a literature search for somewhat older reference materials. In an active research area they are less useful as things get old quickly. This may imply a crossover category, I suppose.