I have to admit to some ambivalence about the issue -- which may be what many of your colleagues might feel as well, and that might affect how they see your decisions.
On the one hand, I think we all sympathize with your situation, and your desire to go back to be a "regular faculty" with a life that revolves around teaching and research. I get that a pandemic is not what you had in mind when you took the job of graduate coordinator, and that the resulting upheaval in all things academic (and non-academic) lead to a workload that is not what you thought it was going to be.
On the other hand, however, you did get a promotion and (I assume) a pay raise that reflects your increased level of responsibility and leadership. By accepting these, you are also accepting the fact that your job becomes less predictable and that you become a more direct subject of the whims of the higher-ups wanting you to respond to this or that. That's really no different than nearly any other promotion: If you work at McDonalds, you can have a low-stress job by flipping burgers Monday through Friday 9-5; just don't accept the promotion to shift manager, because then you have to deal with workers not showing up, unhappy customers, etc. So while you say "this is not what I signed up for", I'd argue that in a sense you did actually sign up for it -- maybe not in such explicit terms, but the uncertainty comes with the job, and it is rewarded in certain ways.
So while I assume that your colleagues will have personal sympathy, they might also ask you why they paid you more (or gave you teaching reductions) if you jump ship when showing leadership is actually required. I suspect that it might also be difficult to find someone to replace you during these times, and that might also factor in how your colleagues see your decisions. A department actually needs its graduate director, so not having one is going to be a problem.
In the end, talk to your department head and your colleagues how they see the situation. You might also discover ways in which you can lighten your workload somehow, and become happier with your role. For example, if you find that during the day, you stomp out the really urgent fires and in the evening at home deal with the mundane and boring stuff, consider whether one could hire a student worker to do at least some of the work that mucks up your cogs. Delegation is often possible, and your department head might recognize that it's going to be cheaper and easier to find ways to make your more comfortable with your role, than to replace you.