The statement claimed he had a leading role in the project (he didn't) and claimed that he contributed significantly to the manuscript (this is somewhat arguable, but he certainly wrote some of it). I signed the document because I assumed that this was just some formality that he needed to graduate, and I wasn't prepared to cause a fuss over the wording in a document that I didn't view as important.[...]However, now it turns out that he needed this agreement because he wants to write a chapter about my work in his thesis."
In order to include joint work in your thesis, many graduate programs require a statement to the effect that you are "a first author" on the paper. So, in fact, that is just what that kind of document is going to be used for. It's entirely reasonable that the purpose of the document was not clear to you (a master's student). Maybe your collaborator should have been more explicit about what it was being used for. Maybe you could have asked.
However, since he is a coauthor on the work, it is also his work, and he is perfectly entitled to represent it in that way. The fact that he needs to be a first author in order to include in his thesis sounds like "a formality that he needed to graduate" to me. Your signing this document does not undermine your first authorship on the paper (I assume you are; you don't quite say so) in any way known to me.
One could put a positive spin on the fact that you, a master's student, have done work that has formed a part of someone else's PhD thesis. That is just about the strongest possible certification that you are doing PhD level work. Maybe think about how to capitalize on that, e.g. in recommendation letters. In summary, so far -- no problem.
Furthermore, he is asking me questions about the paper that are extremely basic. I don't want to come across like I am exaggerating, but the questions he is asking me are the sort of questions that are covered in first-year undergraduate chemistry courses - not the sort of thing that should be being asked by a PhD student about to graduate.
Well, that sounds bad rather than good, but without further context it's hard to evaluate. Presumably this is just a part of his thesis work (you said "a chapter"), so maybe his real expertise is elsewhere? Maybe his background in his field is really spotty or worse, and somehow he managed to do enough work to get a PhD anyway. Maybe he's not so bad but you're really great and thereby have higher standards for what a PhD student should know than others.
Yes, it is also possible that he is some kind of fraud or charlatan. I kind of wonder that you wrote a paper with him and didn't notice until now, but maybe indeed his role on the paper was minimal. My feeling is that while you can respond to this however you like, you have no "whistle-blowing" responsibility here: you are not one of the several qualified, professional academics who are charged with evaluating the student and his work before he gets his PhD. It's on them, not you.
My recommendation: stop answering his questions, and instead say "I think you should ask your advisor about this." He can't lean on you if you don't let him, and you don't need to let him -- as you say, he is about to get his PhD. If his supervisor is also your supervisor, and you have a good relationship with him, then you might want to bring up the fact that the student has been asking you a lot of basic questions. If the supervisor doesn't have an issue with that, that's their problem and not yours. (Even if the PhD student didn't know what an atom was, you still can't do anything directly to stop him from getting his PhD. The advisor can.) If you don't have a pre-existing relationship with his supervisor -- I recommend letting it go.