I am an assistant professor of mathematics (and pursuing in service Ph.D in mathematics ) a non native English speaker. I was not bad at English during my school days. Gradually, my effeciancy in speaking English seems to have gone down badly through years of pursuing mathematics (a dry subject without language). How do i improve myself now with minimum time and effort?
I think minimum time and effort as a goal will defeat you. They are also somewhat inconsistent. To reduce the time (a bit) take a course in speaking that forces you to actually do it. But that takes effort. To reduce effort, just practice as you go along, increasing time.
But the way to learn to do pretty much anything is to practice it and get feedback. In the case of teaching, if you are doing so in small classes, you can just inform students that you aren't a native speaker and are open to being corrected. But that takes a bit of courage and isn't without risk.
If you are in a situation in which you can speak your native language, then try to resist doing so as much as is possible (and polite). Force yourself to speak English in all possible situations.
You can ask colleagues to correct you if you trust them, of course. Again, it takes some courage and patience both from you and them.
If you asked how can I become an Olympic Swimmer with minimal time and effort, the answer would be "You can't". It is similar here.
There is an organization called Toastmasters, which aims to help people improve their public speaking through practice. It's organized in local clubs, which have membership fees (they're usually not very costly). You could consider finding a local Toastmasters club and joining it (note that different clubs may have membership restrictions). The activities usually center around people delivering short speeches and responding to impromptu questions. Toastmasters appears to be international.