Keeping to time is just basic respect, for your audience and for any fellow speakers.
Admittedly, it varies somewhat with context - the only presentation at a group meeting going over probably doesn't have many knock-on effects, whereas at a conference there's coffee going cold outside, parallel sessions getting out of sync, and generally far more going on and more people to annoy.
That said, a rule of thumb: Is your time worth more than that of everyone in the audience combined? No? Then don't finish late - even when they all filed in five minutes late (another pet hate of mine). Yes? No, it isn't.
At the courses and conferences where I've presented, I've always been the last speaker before lunch, where you're already struggling to keep the audience's attention before you start. With a bad chairman, half of that last slot can disappear easily; that speaker is then faced with either making everyone late for lunch or mutilating his/her presentation. Bad chair or not, it's disrespectful for earlier speakers to put other speakers in that position. Admittedly, my experience is that you get massive brownie points for getting back on track and saving the lunch break, and you can get some interesting conversations in the lunch queue as a result, but you shouldn't have to.
Within the department, where we're being kept from actual productive work, I like to count the people in the room, calculate a rough figure for their combined hourly cost, and hence work out how much of our hard-earned funding is being wasted every minute that the speaker goes over. Needless to say, I'm not paying attention while doing this.
The worst example of going overtime that I've experienced was at a conference with a number of lunchtime sessions. Someone whose name started with Sir was assigned one of the first ones, and he was still talking (and his audience still hungry) when the second sessions were over and we were meant to be back in the main auditorium. As it happens, that was the same room where we were booked to give the second session; we ended up giving a software demo standing in the hallway, with one of us holding a laptop for the other and people pressing in to see what should have been projected on the wall.
So, having been on the receiving end of it both as audience member and speaker, my personal tolerance for it is very low indeed. From discussions with many colleagues over many years, I haven't found one yet who's OK with it, even when they find the topic interesting.