I am hosting a podcast related to our field where we invite the stars of our field, the podcast has grabbed an audience from different demographic regions and it becomes very well know in the community. Before that, my ex-supervisor who is in the field and somehow is well known in the field was trying to destroy my career, it was a long story and s(he) was abusive. Now, no one knows about that, I feel sometimes it may sound unethical as I somehow invited everyone the field except him/her. The podcast also got an award from the events organized by the leaders of the field. I don't know how to be neutral, should I invite him/her at some point? because it becomes blatant I am trying to avoid inviting him/her, I know s(he) destroyed me literally, but now we have an audience and at least I have to be inclusive, do you think I have to invite him/her?
I suggest to forget the fact that they are "toxic" on the emotional level, that's between you and them.
You say they are not that inspiring. As organiser of your podcasts, if you consistently invite good speakers, you do not need to invite mediocre ones; and that includes your ex-supervisor.
On the other hand, if you do invite speakers across the board in terms of speaking quality, there is one more thing for you to probe.
You should contemplate whether this ex-supervisor could harm your career if you invite them. For instance, by entering back into your life, by getting influence over you again or by trying to usurp your podcast platform (I have no idea whether this is a possible scenario, but just for you to get the idea what I mean). If that is the case, nobody can expect from you to sacrifice your career just to be magnanimous.
If that is not the case either, it ends up being your decision and you have to absorb the moral dilemma that emerges from it of it. It is a difficult one to make, since you can not just declare "conflict-of-interest" like a reviewer and know that the paper will still be fairly handled by someone else. However, in the end of the day, it is, as CJL says, your podcast.
It would be magnanimous to host them; but do not do this if there is any danger to your career by doing that. And, if you end up inviting that person, keep your interaction with them highly professional, but as distant as you can. You're the boss in this part of town.
Using your position as an organizer to not invite the people you don't like to your thing is the only real reason anyone in academia ever organizes anything. They might retaliate but it doesn't really sound like your relationship is worth anything anyway. Just make sure not to actually say what you're doing, if anyone asks you about it just make up a generic excuse about scheduling.