If I make a formal complaint against my PhD supervisor, where I have evidence of them failing to comply with university policy in Australia, is it reasonable to be afraid that they could essentially try to "blacklist" me (or any other repercussion), as I'm in a close-knit field? Are formal complaints even taken seriously?
Assuming that Australia is like most other places, yes it is reasonable to afraid, as well as potentially dangerous to your career. If someone is not adhering to policy in some area, they may well be someone who thinks more highly of themselves than is warranted and will brook no contrary talk, especially a perceived attack.
In general it is a bad idea to get into a serious dispute with your advisor. They can hurt you in a lot of ways if so inclined.
However, if their breaking policy affects you directly, then you might explore other options than a direct assault to get your needs met. It may be that a "quiet word" with someone would be more safe and effective than a confrontation. A department head, dean, or even another faculty member, perhaps.
But, if you are willing to change advisors and even universities in an extreme case then there aren't likely to be long term consequences, though it is possible for an especially powerful-evil advisor.
The big issue here is that most of the important things that an advisor does are well beyond what is formally required of them. An advisor doing a kind of "work-to-rule" is not going to be an effective advisor for you. If there's really no way to work out this situation directly with the advisor, or at worst some sort of informal meeting with the director of graduate studies, it seems likely that you will need a new advisor. The concern you ask about (getting "blacklisted") is not really the main thing to be concerned about and is much less likely, they're unlikely to advocate for you but it's relatively unlikely that they'll take on some kind of time-consuming vendetta. That said, you do need someone to advocate for you, which is why you'd need to look for a new advisor if your relationship with this one has completely broken down.