I don't want to do my PhD there anymore.
Okay that's how you feel, but reconsider, for several reasons:
A very smart and beloved Physics professor once told me:
Never turn down a job that hasn't been offered to you.
What was meant is that you never know when you might really like a situation, or a specific professor, once you find out more! Appearances from a distance may be totally different than ground truth.
Is there no chance at all that more information might change your mind? What happened to the reasons that made you decide not to go? Are you simply second-guessing your own intuition?
If you feel it necessary, consider a note simply explaining your concerns, and ask if they would still like you to go to the interview. If it's fine with them, then okay you've been upfront.
If that's even a possibility, then consider the added benefit to yourself of practice. In industry where interviews are "just a part of doing business" it's not uncommon to do the first interview for a job (one believes that) one doesn't want very much. It gives the opportunity practice in a lower-stress situation, and to have some experience before doing an interview for a job (one believes that) one really wants.
That will be a bit less common for PhD program interviews of course.
If there are other reasons one wants to skip the interview, then okay. But if the reasons aren't firm, then consider what's potentially lost by not going compared to what's lost by going, and reexamine why you thought you'd like to go and see if those reasons have really changed.