Let's say that Candidate A is at University X, working in the same field as Candidate B, who is at University Y. University Y has lots of faculty working in that area, and an active seminar. Candidate A collaborates with one other faculty member at University X. All the top mathematicians regularly give talks at University Y, which is considered top 5 in that area of academia. Candidate A is hard-working and just as driven as Candidate B but doesn't get to interact with scholars as much as Candidate B. Candidate B has an easier time with research than Candidate A as a result, and publishes more papers of better quality than Candidate A.
When hiring committees or program directors judge Candidates A and B, Candidate B comes across as better, although Candidate B had a lot more advantages. Are those advantages considered when it comes to getting grants or job interviews? Or is it basically a losing battle for Candidate A?