Since you always communicate with your colleagues in English, I'm assuming that "some proficiency" in their language is simply not enough to carry daily life in this language. Having experienced that, I'd be very careful with assumptions, just as others have said, the tone and soem specifics of how he was discussing it can make a lot of difference.
Responding to people in a language they assume you don't speak is also something to be careful of. In principle, people will find ways of talking behind your back if they mean to. You should prefer to have access to these discussions rather than not to have it, but should should be mature to deal with them. That being said, you should occasionally demonstrate that you know a few bits of the language, as pretending to understand absolutely nothing might be seen as a lie. Don't enter a discussion in a foreign language if you won't be able to hold your ground through the whole discussion (which might become a heated one).
Furthermore, if your advisor notices that you have gaps in your education that you need to fill, it is part of his role to communicate this to you, and either provide references your could study on your own or (if these gaps are not that relevant) he should make peace with you having small difficulties here and there.
That being said, think to yourself if this might actually be a matter of "repertory questions", i.e. I can always pick obscure details about a subject you think you are good at and start asking you questions with no previous warning. For people inexperienced in the field, our conversation will make you look like you are not knowledgeable in a subject you actually know a lot about. People sometimes do this deliberately, but also unintentionally, if the asker believes the details to be relevant. The point is, talk to your advisor on whether he thinks those matters he was discussing were relevant or not.