If the sponsoring organization, say ACM, is wise then it will seek a mix of experience and fresh faces on the overall conference committee and subcommittees. This assumes that the conference is part of a continuing series, such as SIGPlan.
The conference chair is generally very experienced and has a wide circle of collaborators, many of which are friends. They choose a few trusted people, mostly experienced, to form the core. Those people suggest others until all slots are filled. The ACM usually (I think) has an organizational representative at the meetings that create the committees to assure that some new people are brought in and general rules are followed. This is to assure continuity for the future so that newcomers get the experience to carry on in future years.
Something similar happens at each major level. The program chair selects a few core people and they nominate and discuss others, with an eye to a mix of experience and new members.
One of the last jobs of the committee for a given year is to start the selection process for some future year (one or two years in advance, say). By the end of one conference the core group for the next (year or so) is largely or completely in place.