I am sorry you are facing this issue and you had an unpleasant, abusive working environment.
One may argue we are hearing only one side of the story: in my humble (and arrogant) opinion, behavioral abuses are fully subjective (like sexual abuses), which means that if you felt you were abused, you were. No need to hear the other side of the story. You are the victim, the co-author is the perpetrator.
Unfortuntely there were no mechanisms in place to have your feelings listened, so I would go as far as saying that the system was perpetring abuse on you, so there are personal and systemic responsibility.
Now, back to your question. The paper is a different system than your lab/department. The authors on a paper are not judged on their efficiency, nor on their behavior. When you say "his bad behaviour also cost me a lot of time.", well, also tutoring a student costs a lot of time, often unnecessarily, however if said student contribute enough to a certain paper, they deserve autorship.
So, on one hand the paper is a different "entity" than your work. I even know of authors lists that carry in themselves couples and lovers, or authors coming from countries being on different side of a war, with all the involved emotional strains.
On the other hand, if you feel really confident, you can think about venting out in the acknowledgements [1], thanking for an unhealthy working environment. It is an extreme measure, but hey, at least your paper is forever linked to the co-authors you do not want but also to a strong sentence regarding their "soft skills". Regarding future consequences for your career: there are few people that read the acknowledgments in the papers from candidates to postdoc&co.
I personally would consider such an acknowledgment as a plus from your side (however I have bad news for you: I am out of the academia, mostly because I hate that universities&co. should set the example, regarding personal respect and freedom, instead they are as bad as they were decades ago and they sacrifice everything in the name of "scientific progress"[2])
[1] be strong, and think about yourself, we are all very brave at typing on a keyboard, but as they say in the Netherlands "the high trees get all the wind" ...
[2] I know, publishing a paper with the co-author is kind of sacrificing all your suffering on the altar of progress, but it is done to set a rule about co-authorship. Every one that contributed deserve authorship, because good or bad judgement is personal (what if your PI decides you behaved badly by putting strain in the working environments by expressing your opinions and exclude you from the paper?)