I was an undergraduate who was paid for his work (rather than receiving credit; I had the option of one or the other, but research credits in the department I worked in were useless to me). I worked at that lab throughout undergrad and am now a staff scientist in the same group.
You really should give your students one or the other. It seems to me fairly unethical to have someone do work for you without compensating them. Of course, there are realities of budgets, but you really should strive to pay people for work they do.
(Honestly, it's ridiculous that you have to pay to take research credits, but that's a different post.)
They also work exactly the assigned hours (says 10 hours/week). You know, like the regular employees in a company, not like a researcher in academia.
Well, yeah. Before I was hired, we set an expectation that I would work 10 hours a week in my mentor's lab. (In fact there may be a limit on students' working hours, my university's was 19.) I don't understand why you think anyone would work hours for free when they are paid hourly. If you need more than 10 hours/week out of them, then hire them for more hours.
Even if you had volunteers; you should set some sort of expectation with them for how many hours (on average) they should put in. Yes, some weeks will be more, some weeks less, but a volunteer should know how much work you want out of them.
Only the students who are really interested in doing and learning research will volunteer to work in the lab for free
This is not true; let me be a counterpoint. Only the students who can afford to will volunteer to work in your lab for free. That is, students who have support from their parents, or who don't already work a job for money will work for you for free. I am sure that the best undergraduate research assistants do not come solely from the pool of people with wealthier parents.
Am I doing it wrong? I'm not cheap, but if paying the students to do research results in wrong motivation and expectation for them, and if not paying them is a good filtering mechanism to select good students, maybe I should do that.
No, not paying them is not "a good filtering mechanism to select good students." You sound like a new professor, and I think your own "filtering mechanisms" will improve as you interview and hire more undergrads over the years. At my lab, we went through about three for a position over three quarters before settling on a good, motivated undergraduate.