I’ll give an answer based on my experiences of academic research in Australia and New Zealand compared with Japan and other East Asian countries. What is expected and what is considered “voluntary” varies considerably and I think these represent different extremes with working in North America or Europe being somewhere in the middle. Of course, individual departments or even laboratories have different working cultures.
In Australia and New Zealand you can freely say no to these things if they’re at short-notice and affect your personal life. Especially if you have a long commute or young family it is understandable to maintain regular working hours. You will be expected to attend conferences, teach undergraduates, write grants, and review papers. However you will usually be given ample time to manage your workload around these responsibilities. Any opportunity to entertain visiting guests or attend seminars is entirely voluntary. Of course this can be great for networking or professional skills but is your choice not to attend.
In East Asia (I think Japan and South Korea) are the extreme here, things are very different. It can be very difficult to say “no” to these things. The working hours are generally much longer and often overtime is considered the only resort to meet deadlines. Due to hierarchical culture, subordinates may not be given much notice. This is unfortunately something that does occur frequently in working environments in this part of the world. This extends beyond research activities and includes participating in events hosted by your institution, hosting visiting speakers, and attending things like the Christmas or New Year parties. In Japanese workplaces, not attending these “social” gatherings after working hours can be considered very disrespectful. These are not “rare” in Asian countries, some labs do this kind of gathering several times per week.
Either way you must consider your situation and what’s appropriate. When applying for a new position I think it’s important to discuss these concerns ahead of time and agree of expectations. If possible talk to other members of the lab to find out what it is like. It varies considerably but does occur to some extend in many research positions. I think this kind of workload only increases in more senior positions with more responsibilities, which is why some is delegated to you.
What you can do is expect that some tasks like this may come up from time to time in the future. You need to decide whether it’s best for you and your working relationships to be assertive and set limits or to allow time for it. Managing your time and workload to be productive is incredibly important in an academic career. Regardless of how common it is, you need to learn to cope with the stress that this causes. I too try to plan ahead and try to maintain balance with my personal life so I understand this can be frustrating.