A good teacher should be more concerned about conveying knowledge than feeding their own ego.
In this case the fact that you are considering moving your lectures for them makes you sound like a good teacher.
I'd say 4/20 is a pretty significant number. Maybe check with the other 16 students for their availability before you decide to move it.
Workshops are shorter and therefore can be opened more frequently. Main lectures for a class are more important. I see this as a lack of or correct prioritization and/or time administration; young people tend to be impulsive, comfortable childish and kind of wild, demand that they make sure the workshop is higher priority and can't be opened some other time.
(1) You may be under contractual obligation to hold scheduled classes. (If so, you may want to point this out to the students if they become insistent that you cancel.)
(2) It may not be fair to other students in the class to let a few students get to, in effect, cancel a class.
I would tell the students that it is fine for them to attend ...
I'm assuming these are not frivolous workshops. In a large university, often one department doesn't really know what the others are doing and schedules are difficult if you want to learn stuff from multiple departments. So if these are "serious" workshops, you can be a bit accommodating. Especially since this is not an overly full class.
You don't want this ...
From the comments:
I provide the notes, worked example and further practice questions on Moodle for those who do, and do not, attend without fear or favour.
If they choose to attend a workshop instead of your class, they are still receiving the material so are not at a disadvantage (other than missing out on what I'm sure is a superb lecture).
They are ...
There is really no need to do this. Lecture is an efficient way for a single person to help a group to learn, but it isn't the most effective way for an individual to learn material. There are other ways to learn the material and they need to become familiar with them. Books, notes, discussions, but most important, exercises to make the material part of ...
If you do this for a minority, then you will have to do it every time a few students have some excuse.
I suggest that you don’t consider this and make it clear that if they miss lectures then it is up to them to catch up on material.
Providing double or triple repeats of lectures due to a few absences, especially if unpaid, is not a good use of your time.