I was an undergraduate who was paid for his work (rather than receiving credit; I had the option of one or the other, but research credits in the department I worked in were useless to me). I worked at that lab throughout undergrad and am now a staff scientist in the same group.
You really should give your students one or the other, or both, if your university allows it. It seems to me fairly unethical to have someone do work for you without compensating them. Of course, there are realities of budgets, but you really should strive to pay people for work they do.
(Honestly, it's ridiculous that you have to pay to take research credits, but that's a different post.)
They also work exactly the assigned hours (says 10 hours/week). You know, like the regular employees in a company, not like a researcher in academia.
Well, yeah. Before I was hired, we set an expectation that I would work 10 hours a week in my mentor's lab. (In fact there may be a limit on students' working hours, my university's was 19.) I don't understand why you think anyone would work hours for free when they are paid hourly. If you need more than 10 hours/week out of them, then hire them for more hours.
Even if you had volunteers; you should set some sort of expectation with them for how many hours (on average) they should put in. Yes, some weeks will be more, some weeks less, but a volunteer should know how much work you want out of them.
Only the students who are really interested in doing and learning research will volunteer to work in the lab for free
This is not true; let me be a counterpoint. Only the students who can afford to will volunteer to work in your lab for free. That is, students who have support from their parents, or who don't already work a job for money will work for you for free. I am sure that the best undergraduate research assistants do not come solely from the pool of people with wealthier parents.
Am I doing it wrong? I'm not cheap, but if paying the students to do research results in wrong motivation and expectation for them, and if not paying them is a good filtering mechanism to select good students, maybe I should do that.
No, not paying them is not "a good filtering mechanism to select good students." You sound like a new professor, and I think your own "filtering mechanisms" will improve as you interview and hire more undergrads over the years. At my lab, we went through about three for a position over three quarters before settling on a good, motivated undergraduate.
After thinking about this for a few days, I would also point out that some types of relationships don't need to be paid. For example, students at my university can do honors projects in their department. Supervising these students is more of a service from professors than an employer/employee relationship, and undergrads seek out professors to supervise them, rather than professors recruiting students.
I assumed from your description of your students having been hired for 10 hours/week that this isn't the relationship you have with them. To put it another way, if you are getting value out of them that either you or a graduate student would have had to perform (e.g. feeding rats, interviewing human subjects) you should pay them. If you put out an ad looking for undergraduates, you should pay them.
In terms of equity, you should always be trying to pay your undergraduates, but I don't want to deny the existence of some relationships where not paying them may not be unethical.