Contrary to the majority of opinions here, as a non-native speaker, I would say that an anonymous email would be helpful. It won't give you any brownie points, and may hurt the speaker's feeling, but it will be helpful to the person regardless. In general, we already give one another too few honest feedbacks out of politeness -- anything in the other direction would be a valuable change.

The most popular answer here asks: "Would you give the feedback if it were about any issue other than English?" I concede that it's not the norm to do so, but it doesn't mean that such a feedback wouldn't be helpful.

Indeed, my advisor gives me feedback on the structure of my talk, its graphs, and its typos -- why not my English as well? (In fact, he does comment on when I need to slow down or fix a grammatical mistake).

The speaker may be aware that he has an issue with language, but he may not realize *the extent* of the problem. There's a chance that the feedback may help, and it sure can't hurt, so why not?

Since you're not as close to the speaker as my advisor to me, I would absolutely advise against a direct conversation. Most likely, it'll make the speaker feel resentful towards you, no matter how helpful your advice is. (Such is regrettably human nature). But an anonymous email would work perfectly well.