If you believe your use is completely legitimate (I'll give an example), briefly state your purpose to the lecturer when asked and be prepared to put away the phone anyway.
Once, in grad school I had a professor tell me to not use my phone in class. I'm hard of hearing and typically record the lectures on a voice recorder or my phone. If I recall correctly I had to interrupt or start the recording on my phone in the middle of the class when the professor spoke to me. I never use my phone in class otherwise, and typically prefer a voice recorder as its purpose is more obvious.
I said something like "Sorry, I'm hard of hearing and I had to turn on my audio recorder." The professor was gracious and might have been a bit embarrassed. There was no problem from that point forward. In fact I've recommended the class and professor to many people since.
If your purpose isn't so obviously valid (e.g., playing a game, etc.) then I'd just put the phone away. In my example, the purpose of using the phone was to better understand the lecturer and minimize disruptions to the class from me asking to repeat things I did not hear well enough, not distract myself.
There are many other reasons which few would argue against, like expecting an important phone call or message.
If you intend to regularly use your phone for legitimate reasons, it probably would be best to talk with the professor before class time about this to make sure there are no misunderstandings. Asking for permission also appears to be required to record lectures in at least some universities.