This question resonates with me, since I am often in what I think is a similar situation. To help fix ideas, I will describe my situation: when I teach math graduate courses -- other than qualifying courses which have a fixed syllabus and prepare students for a later written exam -- the homework issue is often a challenging one. Let me begin by setting the scene: in most math PhD programs in the United States, students continue taking coursework for their entire time in the program. For instance, my department has a regulation that a student must take at least one "real" (i.e., non-reading, non-thesis-writing) course per semester. My own graduate program (Harvard) had the enlightened practice that as soon as a student passed their written quals, their grades in courses are automatically "excused" and thus students do not even necessarily show up at all for the courses they register for (and this is not necessarily a problem for anyone). Most other programs are not like this, and students get letter grades in their courses even while they are writing up their PhD and applying / interviewing for / accepting jobs.
So one is in the position of teaching graduate courses to students, most of whom presumably are at least somewhat interested in the material (they are math PhD students, after all) but many of whom have more pressing demands on their time. On the other hand, for many if not most students attending many if not most courses, simply attending the lectures and never doing outside work is not going to get them anywhere: it would be a more efficient use of their time to simply excuse them from coming to the lectures (which is not unheard of but not guaranteed to be kosher either). So in most cases you want to at least give opportunities for the students to reinforce the material of the lecture, but if you do much in the way of homework then they may be unhappy, and perhaps rightfully so. Moreover, most professors do not get graders for these kinds of courses, and -- more teaching-focused faculty may freely roll their eyes now -- in many cases we have too many other professional responsibilities to spend too much time grading written work. Here are the ways I have navigated this myself:
(1) In all graduate courses my lectures include "exercises" that if worked will reinforce the material. (If I don't do that, then I've lost all pretense of teaching a course.) However in some courses the students are not required to solve the exercises in any way. They are -- of course? -- free to ask me about the exercises, however the last few times I used this practice I had very little in the way of such discussion, to the extent that it would be hard for me to be confident that the majority of the students were spending any significant time working the exercises.
(2) I have sometimes had students turn in written homework, however with the understanding that I cannot grade a weekly problem set in a graduate course. I have a memory of problem sets from an elliptic curves course cluttering up my office and then my study long after the end of the course. I looked at some of them but not all of them and probably not enough of them. I know a very small number of professors that do grade regular problem sets in graduate courses of this kind, and I admire them for it. I know more professors who compensate for this by asking students turn in a ridiculously small number of problems in total, e.g. less than ten for a semester course. This is not a great solution: most of the work goes unread, and they don't get enough reinforcement.
(3) My favorite solution is to have a weekly(ish) problem session in graduate courses, in which we meet -- usually for at least an hour -- and students present solutions to each other (and to me). I like this practice because:
(i) I don't have to grade written homework.
(ii) The fact that students will be presenting in front of others usually makes for more work on any given problem and improves the quality of their presentation.
(iii) When things are going well, it means that students can benefit from solutions to problems that they did not themselves work out.
(iv) It gets students in the practice of talking to and working with each other rather than just talking to me.
I am currently teaching a graduate course (commutative algebra) in which I have "flipped the classroom" by making the Monday lecture a problem session, and then in exchange I give a 60-70 minute lecture on Friday afternoons (of a more one-shot nature; for those who care, my first three lectures have been on: Swan's Theorem on vector bundles and projective modules; Galois connections; and direct and inverse limits).
This is working well -- in fact, better than any problem session I can remember. Whenever I point to a student, they go to the board and solve a problem (one of a longish list that I have given them a week or more in advance). They usually solve it correctly; when they falter, another student steps in to help them out. They do so well that both they and I often feel free to ask followup questions on the spot. Having had the experience of problem sessions that don't work as well in the past, I do not take this success for granted and am trying to figure out what's going on. One of the students is my own PhD student (which doesn't hurt!), and she told me that the majority of the students meet as a group and work out enough of the problems together. That's great! However, the one thing that really makes it work well is that the students are both strong and relatively homogeneous in their abilities and experience: they are mostly second and third year students, and though some are more interested in category theory or topology or number theory, there is no clear top and bottom to the group. There is no doubt that each of the students in the room can solve a positive number of the problems assigned every week.
In past years I've had Kimball's problem: one or two students are -- either by preparation or ability or lack of interest or lord-knows-what -- just not up to the level of the others and the requirements of the course. I have tried to compensate for this in the following ways, many of which Kimball has already mentioned:
(i) Making a wide range of difficulty in problems assigned, and allowing weaker students to solve problems that most of their classmates would regard as trivial.
(ii) Allowing weaker students to present less often than the other students.
(iii) Assigning special problems that target the weaker students' background or are more obviously related to their stated interests. (I can think of one instance where this worked well. But in retrospect I think a big part of the success was that the "weak student" was not actually weak: in fact she was a strong student, just younger and with a worse background than her classmates.)
(iv) Excusing students from presenting solutions to the problems. (I had to do this once, for a student who needed to complete other work in order to stay in the program. This student ended up with a kind of "IOU N problems in subject X" which, of course, was never properly cashed in.)
The bottom line though is that in many math PhD courses, having a student who is not that interested, is not understanding what is happening very well, and who would have to shore up his background a bit in order to engage with the problems at the level of his classmates, is best left alone, possibly after a conversation with his adviser to make sure that he is spending his time productively on other things.