It's normal to advertise PhD defenses broadly in some systems; in many cases defenses are open to the public (at least in part; detailed questioning by the committee is often separate), but as in just about everything in Academia this varies by university/field/country.
It's not expected that everyone that receives such a notice/invitation should attend, but there may be some benefits for you in doing so: you see how defenses are carried out in the department, you learn what your colleagues are up to, you participate in the community of your department.
There are also advantages for the student. I'm in an interdisciplinary field myself, and while I don't understand all the details of research for people in areas further from mine, it is expected that a thesis defense cater to both a broad and narrow audience. Someone who can fit in the same department should be able to get something out of it, even if it isn't narrowly in their research field. PhD students in my program were even required to have one outside person from the department on their thesis committee (not outside the university, as is common in some other areas, but outside their field of interest) to help encourage that breadth of target audience.
I doubt everyone in your department attends every thesis defense in the department, but one good way to find out is to just show up for one. Hopefully you'll learn something new, and maybe you'll have an opportunity to ask a useful "outsider" question. I think it's extremely unlikely you will suffer any direct harm from not attending (that is, I doubt anyone will see it as rude or keep a written tally of who doesn't attend defenses), but you might miss out on opportunity.