It has happened to me the second time that I was turned down by a referee (different one) not taking his responsibility seriously. Of course, I let each of them know about my application and the reference request well ahead of time.
The first time I requested the reference letter I was told that everything would be sent on time, etc. But one week before the application deadline the reference was still not sent (even despite my reminders) and so I quickly found another referee who rescued me. I never told the first referee that I found somebody else and the referee never got back to me over the email again...
The second time I had the "advantage" (or rather a deception) of being able to meet the referee everyday in person and so the day before the deadline I reminded that professor in person concerning the reference, also sent emails, etc. He promised into my eyes he would send it on that very day, but the deadline passed and the reference was never sent... No, the referee did not die, did not have an accident or a bomb in his office. They just could not be bothered.
I think it is very sad to know of the existence of such people and their behaviours, especially considering the fact that they used to be (some of them still are, probably should not be) my close collaborators with whom I spent many hours together. I believed them. I thought I knew them, but only afterwards I realized that whatever smooth words they may have told me in the past, they did not mean them.
How do I ensure that the reference gets sent on time, how can I determine that I can trust somebody who I test the first time?
Usually more than one reference is required and I know that on some people I can rely 100%. Their yes means yes and their no means no. But what about the new referees? I was thinking of telling them of a deadline that is e.g. 15 days earlier than the actual deadline and in case no reference gets sent by my deadline, then I would contact an emergency referee to send the reference by the actual deadline. But sometimes, an electronic application tells the referees of the actual deadline and therefore I cannot make my own deadline.
Also note that many referees do not want to give you their reference that you may read it, but rather want to send it directly to the application committee. Thus something like having a backup letter, I am afraid, is not an option, especially in cases when the reference needs to be signed electronically by the referee oneself.