Short answer: Yes, he can (unless there is a rule to the contrary)
If I understand your description correctly, you are present in the lecture room outside of the designated lecture time, and the professor is enforcing his no-phone rule while you are in that room, but outside of the time for the lecture. Assuming that this is an accurate understanding of your description, I see no reason why the professor would not be able to enforce this rule of ettiquette in the lecture room. It is generally accepted that professors have fairly broad discretion in setting the expected conduct of students while they are in lecture rooms, as they are the designated university employee with operational control of the room. Thus, it is generally considered to be reasonable for professors to impose rules of ettiquette in their lecture rooms, so long as those are not contrary to broader university rules.
In a technical sense, the classroom belongs to the university as an institution, and it is this institution that sets the rules. Your professor is one of the employees representing the institution, and his power is determined by the level of discretion and control granted to him by the university. Professors are generally expected to impose reasonable ettiquette on students while in university facilities, and they are given pretty broad discretion over how they run their classes, and the rooms they are using. Professors are expected to abide by university rules/policy, but within this range, they have quite a bit of discretion. The fact that you describe the room as "his classroom" (though it of course belongs to the university) shows that you have an appreciation of this circumstance.
Now, a professor would certainly have this discretion during the designated lecture time, and in a reasonable window around that time that is necessary for preparation/conclusion of the lecture. When you are talking about periods of time that are far outside the lecture time, there may be a circumstance where the professor loses control of the room in favour of some other university employee, student, or other user (e.g., if someone else has made a booking of the room). Nevertheless, in the absence of someone else having a better claim to use/control the room (and you do not), if that professor is the only staff member present using the room, then he is the representative of the university in control of that room. Absent some university rule/policy that grants you permission to use your phone in this circumstance, or some other more senior person at the university overruling him, it is likely that your professor can indeed prohibit you from using your phone.
I can certainly appreciate why you find this annoying, since "everyone there is an adult". Having said this, universities take in a lot of young students in the early part of their adulthood (many still in their late teens), and there is a huge difference in maturity between these students and their professors. Everyone may have attained the age of majority, but everyone is not equally adult in the sense of their maturity and life experience, so some professors have gotten into the habit of having to play a disciplining role with their students. Many professors find that young students have a tendency to be distracted by their phones in lectures, so some take a hard line. Of course, the most obvious solution here is for you to remove yourself from the lecture hall until you need to be there, and go out into the more general "public" areas of the university, where you are allowed to use your phone, free of the watchful eye of your professor.
