Always stick with the context--your position, their position, the topic under discussion, the location/medium of the discussion. What would you expect someone else in your position to do in that context, if you didn't have any other knowledge of the two people involved? Even if the other person doesn't care what you call them, it's also useful to avoid confusing or offending anyone else around you.
I grew up next door to a university professor and ended up attending the same school--in the same department, even. (Undergraduate level, which does make some difference.) He is older than my parents, but younger than my grandparents. My grandparents address him as Firstname, my parents address him as Mr. Lastname, and I address him as Mr. Lastname at home and as Professor Lastname or Dr. Lastname at school. If I used Mr. Lastname to a student, it would probably take them a minute to figure out who I meant. If I used Firstname in any of those contexts, there probably would be someone listening who would clutch at their pearls (be offended), even though my neighbor himself does not care.
I also work with my significant other's dad, in a workplace where first names are generally used. At work I call him Firstname, at home I switch between Firstname, Mr. Lastname, and "SO's Dad" depending on who I'm talking to. If I used Mr. Lastname at work, it would definitely take people a bit to figure out who I was talking about!