Moderating a panel can be a lot of fun!
First: are the panel members already chosen, or do you get to choose them? If you get to choose them, then look for a group of people with diverse backgrounds and viewpoints. For example, different camps of scientific thought, different career stages, academia vs. industry vs. government, different subfields. Try to pick people who you think will both have interesting thoughts to share and who will be willing to stop sharing them---one blatherer who insists on telling a ten-minute anecdote every time they talk can ruin a panel. Also try to look for panelists who will be respectful to one another, even when they violently disagree: a fight might be "fun" but it generally less productive and interesting than a respectful argument.
It is also critical to be aware that due to the availability heuristic, your first thoughts on who to invite are likely to be biased towards socially dominant groups (typically: white men from Europe and America). To avoid being racist and sexist in your picks, also make a list of people who are not members of this group, and ask yourself who in this list would also be a good panel member. You may be surprised how many qualified people there are with more diverse backgrounds than you thought of at first. Then, from your set of qualified people, pick a mixture that is significantly closer to the diversity that you want there to be in the field, rather than its current average.
Once the panel is running, as moderator, you have three responsibilities:
- Get the panelists to start talking about the subject at hand.
- Keep the discussion at least vaguely related to the subject of the panel at least much of the time.
- Make sure the talkative panelists don't keep the less-talkative panelists from contributing.
A good technique to support all three of these responsibilities is the prepare a list of open-ended questions related to the panel topic. You can start things off by throwing a question out to the panel to break the ice. If the panel starts running down or (badly) drifting off course, you can feed another question into the mix to keep things going on track. Let them drift a while first, though: the purpose of a panel is interesting scientific discussion related to a topic, and drifting a bit may get you somewhere important. Finally, if somebody on the panel isn't getting enough chance to talk, you can inject a question targeted to them: "The next question, I want to start by hearing from Prof. Quiet."
Some panels also are expected to start with mini-presentations, in which case you also have a session chair job, to make sure that the panelists keep their presentations short enough that there is time for a good discussion. If they are giving slides, collect them onto one machine beforehand, so that you don't lose time to A/V changeover.
As for the benefits: sure, you can put it on your C.V. (it doesn't count a lot, but it doesn't hurt). The primary benefits, however, are exposure to the audience, networking with the people on the panel, and the fun of running a nice event.