As he is far more likely than a young supervisor to pass away suddenly due to his age, how should I best plan for this scenario? What will happen to me if it happens?
Well, you will need to find another advisor. You will also need to deal with the emotional trauma of having lost your advisor. It sounds like there is no one else at your university in this field, so depending on how far you are along if this happens, you might switch advisors and move into their field, or someone might take you on as your official advisor and not be able to provide you with much help. If your advisor has several former students who are successful academia, one of them might be willing to be a (possibly unofficial) advisor for you. (If this is not the case, look for another advisor now!) I know cases where this has happened (even if the official advisor was still alive, but perhaps just in very poor health, and deteriorating health may be a more likely scenario than sudden death).
He is also far more likely to retire than a young supervisor – could his retirement (in the unspecified future) cause problems for my future career in academia, with respect to references and networking?
As long as he is well connected and respected now, people will still value his letters and students. But, some people stay quite active after retirement and some don't. Certainly having an active advisor can help a lot after graduation (if they organize lots of conferences they'll invite you often, they may continue to suggest problems or involve you in collaborations, talk about your work at conferences, recommendation letters, etc). It may not be quite the same, but if he has a large group of former students who support each other, probably they will provide a good support network for you.