I think you have to discuss this topic much more nuanced.
Sometimes, people become co-author on a paper where you also collaborated, without you actually knowing them very well or at all. In this case, another author on the paper (likely the principal author) knows both you and the other co-author, both of you did their part, and that's how you ended up being co-authors. Under such conditions, I would not object when you review a publication in which this other co-author is involved. You hardly know them, so you are reasonably objective in my opinion.
On the other hand, if you are talking about people with whom you are regularly collaborating, whom you regularly invite for co-authorship and they do the same with you, whom you know very well, and with whom you potentially have a collaboration going on right at that moment; then this is something completely different. I would never agree to review a paper where any of those people played a significant role in performing the study, and I would consider it unethical to do that, because you can hardly be objective anymore.
Having said that, I must admit that there is still a lot of leverage. You may have ended up being co-author on a paper, where you still have a very professional relationship with the principal author (i.e. they were not members of your work-group, supervisors, etc.) and are still reasonably objective, although you published with them in the past. Additionally, as Captain Emacs already pointed out, some fields are small. At some point you know basically everyone in that field, and may have engaged in collaborations with lots of them at some point. It is in noones interest to significantly reduce the pool of potential reviewers just to stick to some over-the-top rules. I normally decide for myself if I can be objective in regard to certain authors, and decline to review if I cannot (and inform the editor if I know any of the authors on a significant level). And I just hope, other reviewers are doing the same.