Imagine their situation. If you were a member of a native English speaking team, would all of you happily switch to German at lunch time, only because a single member of your team speaks German better?
Would you do it happily in your own country?
And, from an ethical aspect, in my opinion, you simply can't ask this from them.
I also live in Germany as a foreigner and have lived there for years. I can talk with the Germans seamlessly (both in English and in German; mostly we use German), but the lunch times are somehow still out of my reach. There, they speak fast and often dialectic between each other. This really isolates me and this is really annoying.
But, I accept it and try to do my best. I accepted this as I first stepped over the German border with the intent to remain there for a long time. In my opinion, you should also have accepted that.
(By the way: our problems are nuances compared to the problems of people trying to live in France.)
