I don't think there is an ideal. Lots of things can work if people are committed, though continuing commitment will be an issue for most.
I'd suggest asking a professor for some guidance, especially initially. They can suggest books and papers to read.
More than six in a group will start to get unwieldy. Two is fine if both are committed. But the process will differ depending on group size. I'll suppose closer to six than two in what follows.
Someone should take the lead in scheduling meetings but that is probably enough.
One way to make it work is for everyone in the group to study the same material for a week (or so), but one of the members (rotating) is tasked with making a presentation to everyone on that material. That can include both an oral presentation and a written summary. Everyone else is expected to read and actually put some effort into the same material, but that will probably be the hardest part to maintain.
But someone needs to be the "leader" for any given meeting; the presenter. If no one leads then little is likely to happen.
The presentation of the material includes a discussion of it with corrections suggested. To really do a good job of it the presenter then produces a second written report that includes updates from the presentation. This is distributed to everyone.
People could volunteer for some set of material or the "organizer" can assign people, say in round-robin fashion.
Let any of the members suggest new material. Check back with the professor on occasion, perhaps by showing them the reports.
Since this is math, don't neglect the fact that a lot (most?) learning takes place while doing exercises. A group can be used to get feedback on individual attempts at problems - especially difficult problems. It is easy to be misled as to your level of knowledge if you don't test it with application to problems.
Note that this isn't limited to students. A group of professionals can do the same thing.