In order for your articles to be published in a good mathematics journal, they will have to be peer reviewed. Peer review followed by acceptance is the only real mechanism that will lad to anyone taking notice of your work. Serializing a book this way may give you a good chance to get traction with the mathematics community and to validate your ideas. Just putting a book up on your website is unlikely to achieve any of your goals because the community has no reason to trust you. Peer review is a mechanism to weed out bad ideas and highlight good ones. It's not perfect, but it is frequently helpful in this regard and is the way the modern mathematics community works.
Publishing your work as a book on your website could be an obstacle to publishing it in a journal. You generally have to transfer copyright or give some sort of license to the publisher that guarantees them some way to make money off your work by selling offprints or downloads of your articles. Some journals don't like it if there's any preprint versions floating around out there that are too close to their version. Fortunately mathematics is a heavy user of the arXiv, and so certainly pre-review copies on there (or Github) are less likely to cause a problem with the journal. Read the author agreement and copyright transfer/license requirements of the journal you'd like to submit to before putting things out on the web. The arXiv is well known in the mathematics literature, so you'd probably have better luck, more visibility, and less problems posting your articles/book there instead of Github.
BTW, LaTeX errors don't usually lead to rejection, resubmission, and re-review, they are usually just fixed after review. Hopefully your article does get published in the end, but something seems fishy about the details of your story.