I'm not familiar with conferences where you do both an oral presentation and a poster presentation, but I'll share my experience doing posters.
The most important thing to remember is that everyone there knows how posters work.
Everybody expects to wander up to an interesting poster and maybe get the last half of an explanation. People who are genuinely interested in your research will wait for you to wrap up with the last group before asking you to start from the beginning.
I also find that most people are pretty respectful of a group of people waiting their turn. If you turn around, and there are three people waiting to hear the talk from the beginning, it's polite to cut your questions short and let them have their turn.
Just keep a smile on your face and go with the flow. Depending on the size of the conference, this might never come up, or you might be constantly managing three or four attendees who are at different stages in the presentation.
One solution I think is: Wait for few minutes, let a bunch of people present and then start presentation
Like I said in the comments, this is silly. If you meant "let a bunch of people present [themselves]," i.e. show up to your poster, no one is going to stand around and wait for you to get a critical mass of people. That sounds painfully awkward and people will just wander away, especially if it's a short poster session.