First of all, I'd like to express my opinion that none of us here have the moral authority to tell you what to do, and you should be suspicious of anyone telling you that you definitely should or should not do this or that. This situation is very serious and you are the one who will be living with the consequences of your actions, not us anonymous (or not) internet strangers.
Instead of telling you what to do, I thought it may be helpful to list the choices you have and each one's pros and cons. Here they are as I see them:
1. Continue to investigate by sending a fake application to the job posting.
Pros: through such an investigation you will be helping (if your suspicions prove correct) to expose a very serious ethical breach (bordering on a criminal offense of conspiring to commit fraud) by the student in question, and helping to rid your department, university, and the academic world of someone who clearly has no business being there.
Cons:
You will likely incur the wrath and hatred of the student you will be investigating (and possibly his family members, friends, and even other grad students in the program) later on when he finds out you were the one who sent the fake application that helped expose him.
You may very possibly be suspected of sending in a real application, which would implicate you in unethical behavior yourself and cause you to get in serious trouble. Make sure to document your activities in a way that clearly establishes your honorable intentions. Even then, you could end up being accused in some unexpected way of causing harm or even doing something illegal by your meddling.
By choosing such a high level of involvement in what sounds like a very messy affair, you may cause yourself a lot of wasted time and emotional entanglement later on (e.g., being debriefed or interviewed by university officials and administrative investigation committee, even having to testify in court some day).
The bottom line is that this course of action carries a significant amount of risk and potential for trouble for you.
2. Continue to investigate by asking the student about his activities as one of the answers suggests.
Pros: can't think of any. Someone so immoral would almost certainly just lie and you will gain no information.
Cons: by asking him what he's doing you will alert him to the fact that his current deception scheme is too easily detectable, making it likely that he will come up with a better, less transparent scheme, and ultimately helping him to defraud the university.
3. Do nothing, just ignore what the student is doing and mind your own business.
Pros: no work for you, no wasted time and emotional entanglement in a messy scandal, no colleagues who hate you for getting them expelled from school, etc.
Cons: you will have to live with the knowledge and potential guilt and shame associated with having known about the student's possible unethical and maybe illegal behavior and done nothing. The student will go on to fraudulently receive his PhD and your university'a reputation may suffer as a result. Your own degree may be worth a little less as a result. In a small but real way, all of society will suffer.
4. Report your suspicions to the chair of your department and/or the student's advisor and/or other appropriate university officials.
Pros: you don't become involved in the affair in a messy, major way, but will likely lead to the student being exposed if he is in fact guilty. You will also know that you did the morally right thing by reporting the student and won't have to live with the guilt and shame of having done nothing.
Cons:
you may still eventually become known as the person who helped expose the student (you can try an anonymous complaint if you want to keep yourself completely out of the story, but I think that will be less effective and would make it harder to prove the student's guilt), with the possible animosity and other negative consequences I described above. However, the level of animosity would likely be less great than in the scenario where you submit a fake application.
you won't get the satisfaction, excitement, and superhero feeling that you might get by becoming actively involved in the investigation and playing private detective as in the suggestion to submit a fake application.
To summarize, you probably want a recommendation about which action to choose, but as I said, I don't think it's right to offer one. You will have to make your own decision, but hopefully the analysis above may still be helpful. Good luck!