I assume that the conference attendees will be at a higher educational level,
which probably means they can not only listen to talks in English, but probably are also able to communicate in English.
So yes, I thank it's safe to go to the conference and you will be able to talk to people in English with no big problem.
I guess from your reservations that you are aware that it is not always easy to get along with English in Germany. Don't expect sales agents or bus/taxi drivers to talk English. And even if people speak English, they might not be very fluent.
And I must admit that all other participants being German is strange - whatever reason there is that talks are in English, it is surprising that participants do not come from outside Germany. I'd propose to call the organizers just to make sure that talks are in English.
I don't think you're paranoid here. E.g. Universities will have lectures announced in English, but hold them in German (as long as nobody objects).
And if there are so many German speaking people, groups often will talk in German. Then you shouldn't be shy - just throw in a remark and make clear you only speak English. I'm pretty sure they will switch language then.
Note: I'm saying this as a German living in Germany, with some experience with conferences at different national levels, sometimes visiting conferences where most participants are locals not speaking German. In the last case, it's really important to not be too shy - when you try to join a discussion, make clear you do not understand what they say as soon as you get the chance - maybe a simple "hello, how are you" is already enough to achieve awareness.
I guess you will get to know some nice colleagues.