I want to put the LaTeX source files for a paper that I just published up on GitHub (technically they are already there, but private). Is it inappropriate to include the (anonymous) reviews and my responses to these?
Yes. A review is normally a job performed by somebody at the request of a journal editor for your benefit. Hence the review is not yours per se to do as you please although you have received it to improve your manuscript. The review is basically a communication between two persons, albeit through an intermediary. Thus posting somebody's comments without asking for permission is similar to publishing somebody's private letters online without asking. It is not strictly illegal as I can see it but it is inappropriate.
If reviews are done in an open system, they are public and then the problem is a non-issue. Such reviews, if one wants to couple them to a paper or manuscript online, can, and probably should, be linked anyway.
As a side point, some journals only allow you to post pre-review versions of your manuscripts online. Revised manuscripts then involve the journal's "services" and are apparently seen as covered by the publisher's copyright rules, at least until they are rejected. This is not something I find appealing, it is just the case and one needs to take a close look to the agreement entered when submitting a manuscript.