Is it that the people who land the jobs are harder working, or better at networking, have more famous advisors, have more prestigious papers, or something else?
All of those things. A good economy. Lots of retiring old professors. Luck. An open and friendly personality. Some obvious "fire" in your gut about pursuing questions. Teaching ability. Flexibility. Luck. A wide search. Oh, and Luck. Ideas. Insight into your field. Willingness to share. Did I mention luck? Persistence. Being able to speak, even if you find it difficult as an introvert. Supportive colleagues/advisors. Willingness to do more than the minimum asked.
And, luck, of course.
None of the current answers have mentioned small liberal arts colleges (SLACs), so I'll add a few things from this perspective.
I am a tenure-track faculty member in mathematics at a SLAC in the US who was on the job market in 2017. Although some of the items mentioned above definitely affected my application process, here are some other factors that I saw have a big impact:
I was very deliberate in practicing my teaching. I served as an instructor of record (not just a teaching assistant) for multiple courses while in graduate school, completed a certificate in college teaching, and tried to TA a wide variety of courses.
I was meticulous in the preparation of my application materials. The graduate college at my university hosted a handful of workshops on the subject and even offered individual career consultation, both of which I used.
Despite being somewhat introverted, I was careful to present myself during interviews as enthusiastic, energetic, and excited about working with students, faculty, and even administrators. (It's important to note that this was honest, too! It was all about marketing myself as effectively as I could.) I did this by asking targeted questions, presenting my own ideas, and more.
These things didn't just help me! I found that my colleagues who were also successful in their applications to academic positions tended to be strong in these areas as well.
I think this is still on topic, but I highly recommend Karen Kelsky's book, The Professor Is In. It was an easy read and invaluable to my application process. Along with providing useful advice on subjects both large and small, it also helped to make me a much more confident applicant--something whose value should not be understated!
Something that hasn't been mentioned is a lack of financial motivation. Academic jobs, especially for early career researchers (and "early" here can be as late as mid 30's), tend to pay less than the private sector. There's a lot of people out these who probably could have stuck it out if they'd have been willing to take another temporary research post with middling pay and an uncertain future after that. But for people who want to move into a bigger house, start a family and so on, the private sector usually offers a better paid and more stable alternative in those early years, if perhaps a less interesting occupation.
The obvious response is
Gender and race
The fact that it is not the first thing coming up from academicians is a clear indicator how this is a poorly understood issue.
Please have a look at the example of (ex-)professor Christopher Jackson, who tried to bring up the (lack of) diversity topic in the rather conservative world of geoscience, receiving only backlash and being totally unsupported by the institution he was affiliated, or the statistics coming from Germany, where the 35% of completed PhD identifying as women are matched by a ~20% of people identifying as women have the professor title.
Demographics of completed PhDs and Postdocs (or Assistant Professor) by race shows similar filtering out of ethnical minorities in the US, with the exception of the extreme increase of people of Asian ethnicity doing more PostDocs (yes, one may say that Whites are discriminated to do a Postdoc but ... one should check what will be the resulting distribution by putting in the same bin PD and Assistant Professor, both a Postdoc and Assistant Professor position can be taken afterwards a PhD and rarely one would prefer a Pd to a Asst.Prof...).
- Consider a job in academia to be a good thing (many who work outside academia deliberately choose that path as it is preferable for them, or hunt across both and see where their career goes). [the below don't apply to all those who don't want a job in academia, frankly these also apply to getting jobs you want in pretty well any industry]
- Luck - jobs are advertised in areas that align with their skills, at a time they are looking, in locations they are willing/able to live. Are the jobs you want in places you have a legal right to work?
- Expertise - in the field/topic, in writing grant applications, in writing papers, in supervising students, in teaching. The most competitive jobs outside academia are every bit as competitive (or more) as those in academia. To get into highly selective jobs you do have to be good. Talent does matter. If a place receives 50 employable applications with only 1 position available, it's not enough to just be good you do need to stand out.
- Your network and how well they have worked for you. The concept of sponsoring is important here. Some people are given opportunities (usually because of their network) unequally to others who work just as hard, are just as talented etc. Sometimes they get these opportunities because they know how to make themselves stand out to people who can offer them, other times not so much (bias and privilege are real and play out here strongly). These opportunities often lead to outputs that make a candidate more competitive. This can be everything from inviting someone to be on a grant, bringing an undergrad onto a paper, hiring someone to do a small amount of work in your lab, being a supervisor, funding conferences/travel experiences etc. Without hard work these opportunities get you nowhere, but you can definitely capitalise on them. These are also all things that most people cannot offer equally to everyone. I can offer every student equal access to a resit of an exam, I can't supervise every student who applies, hire every student who wants a summer project, help every student write their first paper etc.
- Your own efforts, ability, and willingness to work the political side of the system and make it work for you.
Academia really isn't some special workplace completely distinct from all others [I've worked industry, government, and academia professionally]. There's some things that are differentiators from other sectors (true of all sectors/fields) but half that battle is understanding it is just an employment sector with pros and cons. This question reads more about what differentiates people who get jobs in academia than those who can't despite trying than about what's different between people who choose academia vs those who choose any of the many other options - the latter question I feel varies a lot more by discipline though and is harder to answer.