In choosing between PhD programs A and B (for pure math in the US), some professors have remarked to me that A is favorable because it is more selective and consequently has stronger students than B. Is this a factor I should consider? Both institutions have several of the top researchers in my subfield so quality of advisors is not an issue.
It's worth considering in your case because your graduate experience will in part be shaped by your peers. Your peer group in graduate school will help (or hurt) your research and teaching experiences --- at some schools your peers will help lift you up and inspire you, at others they will underwhelm you.
Your peers will also help with networking. The people who are finishing when you arrive in grad school will be useful people to get advice from when you go on the job market, as they will have recently been through it themselves.
I'm in the finishing stages of a PhD right now and I've benefited a lot from knowing the top-notch students who had come before me. I learnt from them, they have given me lots of feedback over the years, they've helped me with networking, etc.
Most people would say that a degree from a more selective university is more valuable.
Most people would say that high-quality peers is a great thing to have.
Neither of these things is more important than your degree of fit with prospective advisers. Neither of these things is more important than your personal motivation and happiness.
Find a grad school that makes you feel really good about your studies rather than maximizing the future value of your degree.