Oh boy, is there a lot of cynicism in the comments ...
Is it ok to say no to such an offer and explain why I do not want to go, or will this be carreer suicide?
First off, saying no to one specific conference invitation is certainly not "career suicide". Opportunities come and go, and all of us sometimes (or rather: quite frequently) need to leave some opportunities on the wayside because of circumstances. So if you feel particularly bad about this specific conference then just say no and go on with your life. Sure, just maybe this would have been the transformative event for your young career, but much more likely it would have been like most conferences - mildly interesting, but not something very different to the dozens of other conferences you have or will attend.
That said, saying no to all international conference invitations will be a severe hinderance (unless academia radically changes in the next years). I understand your concerns, and they are definitely valid, but right now academia is a reputation game, and not attending conferences would be a disadvantage you likely can't afford. We all like to believe that our journal papers will speak for themselves, but my slightly cynic observation is that they do not, if the competition also has good journal papers and travels around the world to advertise them.
So the question I would ask myself is: is this conference much worse than other conferences in terms of climate impact? For example, if you live in the US and the conference is in Australia this may be the case - there will be plenty of high-profile conferences that you could attend instead, some of which you can take the train to. However, if you live in Australia, almost any (international) conference will be "halfway around the world", and, quite frankly, you will need to come to terms with the idea of going to such meetings once or twice a year.
One way not to reduce the climate impact, but to make it more "worth it", is to combine your long international travel with other activities. For example, if you already go around the world you can take the opportunity to travel, have a nice vacation, or visit other universities. When I was a PhD student, I travelled to a lot for conferences, but I never went on a plane just to take a vacation (and even now, 10+ years later, my desire to take a long flight just to see a place is limited since I have visited most places that interest me after a conference).
What is the best and most diplomatic way to decline without causing too much damage?
If your group leader is sympathetic to your climate concerns you can just tell them. If not, I would make up some other reason (concurrent obligations, some family commitment, etc.). As you can see from the comments, the sad reality is that some people will get really upset if you bring up climate concerns, and start questioning your commitment to academia. If you are unsure I would probably make up a white lie.