You need to decide is how formal you want to make this meeting. At the moment it seems like you are in a situation where people are treating it as a casual event and hence the attendance is patchy, whereas you would prefer consistent attendance and engagement. If you want that, you need to make sure people see the event as a formal, scheduled part of the department's activities.
The first thing to do is add the meeting to the department's calendar. Furthermore, advertise it regularly via email (a week and a day prior to each meeting, perhaps). Depending on the subject matter under discussion, you could even advertise it more widely (to other, neighbouring, departments, for example). Another way to attract attendance is to get a senior member of the department, perhaps the head, to regularly attend and participate in the discussion. Provided this person carries the usual amount of respect amongst their colleagues and students, this will encourage other people to attend, as they realise that the department head is taking it seriously and hence they should too.
I would not remove people from the group if they don't attend. We're all adults, and the workplace isn't like an internet forum with a moderator to block certain users. Removing someone is only going to dissuade them from ever attending again in the future. It could be that they are just not that interested in the book you are covering at the moment.
In terms of selection of material, in my experience it is best to make a pre-selection (perhaps choose four books) that the group members can vote on. People will be more engaged if they have had a say in the selection process. Otherwise it just feels like a class at school with homework imposed by the teacher. But, don't make this a drawn out process. Send round the list and give everyone a week maximum to vote, and then send out the choice immediately after. Taking a long time over a selection will only move the reading group further down people's mental priority lists, again leading to lower attendance.
Finally, I would suggest making perhaps a single slide to summarise the book you have read and posing two or three questions to guide the subsequent discussion. Keep this general theme in mind as the discussion continues, and don't be afraid to steer people back to it if they go off topic, provided you warn them at the beginning that you will do so. This will also help with time-keeping: it's important that you don't let the meeting over-run, as this again will discourage people from attending in the future.
My number one piece of advice would be to enlist someone to co-moderate with you. It helps enormously to have someone to shortlist material with, as well as to hold each other accountable when completing the reading. Finally, and most importantly, having two moderators means that you avoid any awkward silences during the discussion, as the two of you can mention and answer the prompt questions you showed on your slide at the start.
