In testing sessions that I proctor now, there are frequently one or more students who are not actually working on the test for extended periods of time. Perhaps they are just staring into space, or out the window, for most of the period. Or they may be holding the test paper up in the air and apparently scrutinizing it for a long period of time. Or a student may lean back in their chair and go soundly to sleep.
Excepting this last case, the student likely only has to adjust their eyes a few degrees to be looking at another student's paper. It gives all appearances that they are watching for me to turn away so that they can cheat. I feel compelled to watch them constantly for the entire period, which is tiring and stressful, and doesn't let me watch other students or answer questions. Frequently, the paper I get back is mostly blank after this hour-long staring match, but I can't know that in advance.
What is the best way to deal with inactive students in the testing session? Is it appropriate to demand that they leave the testing area if they're not actively working? (Students are already told in advance that they can turn in their paper and leave as soon as they're done with it.)
I'm teaching mostly lower-level mathematics at a large urban community college in the U.S.
Added: This is in a moderate-sized course with around 25 students in the classroom, taking the exam. The students are known to me and registered for the course which I am teaching. I made the exam and distributed a practice version, with identical directions for each question, in advance of the exam.