I think that if you carefully consider the whole set of things you've said, you'll see that you already know the answer.
First, there is the negative situation that you want to stop:
- "Students are taking advantage of my generosity"
- "Students nearly always stay well past the point when my office hours are supposed to end, even a couple hours worth in most cases"
- "The main issue here is that these appointments often last much, much longer than an hour"
- "Their behavior is seriously impinging upon my ability to progress in my research"
Second, while you do want to solve this problem, you revealed some results you don't want as you solve it:
- "I don't want to discourage their eager approach to their work"
- "I don't want to directly shut them down"
Third, there's what you've tried so far to get them to use less of your time:
- You have office hours (presumably posted) that state an ending time.
- "Subtly imply that I would appreciate less consumption of my time."
- "Deliberate scheduling of an office hour immediately before the taught class"
And finally, it seems like your goal is:
- "Compromise between my need to progress in my work and my desire to assist my students"
Frankly, it looks like the real problem (to me) is that you want students to leave when they've taken up a reasonable amount of your time, without you having to directly tell anyone it's time to go.
But you have to ask yourself, why on Earth would they do that? All you've done is bend over backwards to accommodate their use of your time at every turn! And even if you had been more clear, there is a difference between what's reasonable and what's realistic. It may be reasonable for them to value your time more (by using less of it), but it's completely unrealistic for you to expect that to happen, without you doing something to make it happen somehow. People are fundamentally selfish and will use as much of your time as you don't stop them from using. It's up to you to stop them.
You want a compromise without changing your own actions, but that's not a compromise. Part of the compromise you have to make will be giving up some degree or quantity of your apparent wishes/goals in order to get others that you value more. Specifically, you may have to disappoint some students from time to time, and until you're willing to do that, you'll continue to feel like a doormat and be used like one.
It basically comes down to this: You need to decide what you want, and then make that happen. You don't know what you want right now, besides wanting to not have to deal directly with the problem of students taking more time than you wish. Except... I think you do know. You want the problem to fix itself, but know deep down that it won't. What you're afraid of is true: you have to do the hard thing and set limits.
Also, I would like to suggest that setting limits on student use of your time is not "shutting someone down" or "discouraging eager work". Saying those things is, in my mind, just a means to avoid the unpleasantness of having to tell someone to leave and being firm about it. If students need more help than you can give, that's not your problem to solve for each and every student. Live in the real world, where you have a limited amount of resources and you must apply them wisely. Your resource of time is being misused, and you must stop that. It is not fair to you to be used all the way up. Students have to study by themselves eventually, and you need to give them a reason to do that.
Here are some ideas.
Communicate. Actually tell them what's going on and ask for help. Hold a meeting with all the students, or print up a flyer, or put each student's name in a book and check it off once you've communicated this. Tell them, "Dear Student, I love helping all of you, and wish that I could continue helping you in the way I have until now, giving you all of my time and energy. However, as much as I hate the idea of leaving you hanging, I have to reduce the time I'm spending, and from now on I'll be ending office hours at the stated time and appointments after 1 hour." Ask for them to brainstorm with you how to fix the situation. Ask for their help directly. Ask how they actually want to grow as people and if just relying on you is their idea of true personal growth? Whatever you decide on and you clearly say will happen, do it.
Delegate. Brainstorm with the students to find ways to use your time more effectively. Suggest they set up their own study groups. Get the more advanced/smarter students to tutor the less so. Assign a student to come and kick people out for you.
Be consistent. Do make use of strategies for avoiding the confrontation of saying "it's time to go", but don't use only these strategies if that leaves times when you don't make someone leave. You have to change the way things work in order to set clear expectations. The moment you let even one student stay late, you set up expectations that are at variance with what you want, and that is truly an uphill battle. Students have to know you stinkin' mean business about ending at the right time, and the only way they know that (no matter what you tell them) is by actually doing it. Every time.
Demonstrate. One strategy that can help you is to use non-verbal signals. Get up and walk away when something is over. Even if you can't leave the room, can you get up from the study table and walk back to your desk? Can you take off your glasses, turn off a light, go into a different room, close up books and put them in packs? Get a neon light that you turn on when it's consultation time, then turn it off when it's over. Go open the door and stand there holding it, looking expectantly at them. Look obviously at your watch. Set a timer or alarm that rings (use a 15-minute warning if you like, too). Start talking about next time. Act as if you don't have to tell them they need to leave, act as if they already are in the process of leaving, and you'll find that they get the message. Physically break eye contact, turn your body away from them, and take a few steps. Leave the room and walk away, asking them to make sure the door closes when they leave. All these can be easier than having to say "it's time to go now." I promise they'll work for almost every student.
Be firm. By this, I mean precisely one thing: don't discuss it at leaving time. Discuss the new rules all you want, but never at leaving time. Leaving time is for leaving. Talk about it next time you meet. Make an appointment. Send an email. Write a reminder note. But get them out the door.
Be oblivious. Use this as a last resort, but for any students who still have trouble leaving, and you truly still feel so uncomfortable repeatedly saying "see you at our next meeting" or "I really have some work to do" and really making them leave, then simply act as if they've left. Ignore them. Get busy with your own work. When they impinge on your awareness, act startled, and say "oh, are you still here? Office hours are over." Then go back to your work and ignore them. It will work. You don't even have to tell them to leave.
Give yourself a break. Recognize that you are the only one who can decide what you want, and you're the only one who can make what you want happen. If you don't choose to make what you want happen, you're the only one to blame for that (I guess you didn't really want it, enough, after all). So don't beat yourself up for not being Superman. You deserve to have the kind of life you want (within reason). It's clear you love your students, but it's not loving to them if you burn yourself out, or come to resent them over time.
Get real. You may not actually be helping your students all that much by being on call for them all the time. Will they ever learn how to study on their own? Are they learning the right habits that will help them in their future studies and careers? At some point, you have to stop flying your babies around and let them use their own wings.
Important note: I'm cognizant there is tension between be consistent and be firm on the one hand, and be oblivious (and possibly, demonstrate) on the other. However, realize that we're not dealing with a perfect world full of superhuman gods and goddesses—we've got a somewhat harried yet good-hearted T.A., a mere human, who struggles with setting firm limits on people he or she otherwise loves to serve, and who doesn't exactly relish confrontations that result in their disappointment. Since we're living in the real world, instead, I offered both what I think should happen (be firm and consistent) and also threw out a rescue line for situations where a particularly insistent or oblivious student makes things difficult. Think about it: once the lesson is learned (by the T.A.) on how to help people leave, and the experience is gained, even using the be oblivious "game" I suggested, change will occur within, and in time, this game won't be required. Consider: in utilizing these strategies, the O.P. will more likely be successful at the firm part, because in no case will he or she continue assisting the student past the requisite time, and that's the important part.
Don't assume others are exactly like you, and please recognize that sometimes, less-than-perfect coping strategies are necessary. For people who can stand to grow and are finding it difficult, they may need baby steps. Have some compassion!