Acceptance rates are nearly meaningless as isolated numbers, for exactly the reason you describe. They are averaged over the entire applicant pool, and therefore they imply very little about any particular applicant's chances.
If you are trying to estimate your own chances of admission, then you need more information. If you attend a school that regularly sends students to a given university, then you can form a decent estimate based on feedback from faculty and comparisons with your peers. Otherwise it's harder to predict. At top grad schools, it really depends a lot on what your recommenders say, how seriously the admissions committee takes their letters, etc. You may have a faculty mentor who can help you arrive at a good estimate, but you'll get at most a crude approximation from grades, GRE scores, counting REU publications, etc.
So is grad school really that hard to get into (If you are on top of your game)?
For people who are genuinely on top of their game and have made sensible decisions about where to apply (based on their accomplishments), getting into grad school is not that hard. They might not be admitted to their dream schools, but they'll be admitted somewhere reasonable. On the other hand, thinking you're on top of your game doesn't imply that you are.
This makes it difficult to give universally applicable advice. Some applicants should stop stressing out and have faith that it will work out OK in the end, while others need to start worrying and rethinking their plans. We end up back in the same situation as with admission rate statistics: it all depends on your personal circumstances.