Make sure that if you are going to do this, it leaves him enough time to find a new group (assuming this is related to a class or an undergraduate/masters project and not a phd collaboration). Going back on your word is one thing (subjectively unethical), but it would be extremely unethical to go back on your word if he does not have the time to adjust to your news accordingly.
If you two have already formed a project idea together, view every idea you have told him as his own. Absolutely do not take anything from your current project idea to the new team (especially if he is unable, or too lazy to, find a new team that has their own idea)