If you are invited to give a departmental seminar and the department offers to put you up for up to two nights, how long should your visit be? I find it a little rude when our speakers arrive an hour before and leave an hour after the seminar. Do you have to spend a full day at the host department?
In my opinion, it is not rude to make only a brief visit, especially if you accept with, e.g. "I would love to come, thank you, but I'm afraid I can only come there for the afternoon - I hope that is okay?"
Certainly I agree that it is as great idea to stay for the whole day if you can.
I've noticed that perception of etiquette seems to differ a lot from person to person. Whether or not it is okay to recommend a taxi from the airport (as opposed to picking up the visitor yourself), whether or not to take visitors out for a late drink (or karaoke!), whether to put them up in a hotel or your house, how long to encourage them to stay... these are all aspects where I've seen things done differently. But a small minimum of communication beforehand and flexibility during the visit seems to suffice to guarantee a good visit.
If you're the one visiting, I would certainly recommend at the very least hanging around the department some and going out to dinner with your hosts (assuming they invited), but it's more or less up to you.
Think of this as an opportunity, rather than an obligation.
Find people in the department who are doing interesting work and try to organise a short meeting with them. Volunteer to hang around after your presentation to talk to PhD students (and ensure that your presentation has some appeal to keep people around). Ideally, try to have a fairly full schedule to maximise the benefits gained from the opportunity, though avoid having every minute planned so that you can have spontaneous extended discussions with people, should the opportunity arise.
Planning ahead is probably key, as not everyone will be able to accept an unannounced visitor for a lengthy discussion.
People who give invited seminars are quite busy. It is frequently the case that they may have to come late or leave early. However, unless the guest is local, it is usually considered normal for a guest to spend a day visiting the department. The reason for this is that many times there are people who wish to meet with the visiting speaker, and thus an extended schedule is necessary. Cutting the visit short may also deprive you of meeting people who might valuable future collaborators for your work efforts.
However, in circumstances where the visit does need to be curtailed due to length, the more important it is to communicate this with the host organization in advance.
If you are the one organizing the trip, telling the visitor that you are covering one or two nights' stay is enough of indication that the department wants to host a long enough visit that would allow meeting with other faculty, post-docs, and possibly grad students. If the visitor does not take the hint, you may have to ask them, "So basically you are refusing to meet with our department?" -- and you can actually take it to the chair and ask whether it would make sense to withdraw the invitation, as this visit is the event that is supposed to benefit the department, not just hit the financial bottom line. Basically, whoever is paying for the trip should be fine with an abridged visit. If the department is paying for it (the round-trip ticket will be a fortune for the same day trip), it has all the rights to have the visitor to themselves to ask questions and promote the young researchers of the department. If somebody's grant is paying for the trip, then let have them waste the money. I ran a department colloquia series for a year, and never had an issue like that with any of the visitors, including some biggish wigs. (We are sort of ways away from big airports in a US midwestern college town, so there is little physical opportunity for the visitors to escape; it's not like you are taking a train ride from one university in Boston to another.)
If you are the one visiting... well, if you don't take the hint, you will lose respect of that whole group. If you are a young researcher, that may be a big hit to take: people talk, and in a year or so, the rumor may spread to half of your discipline about your rudeness. If you are a named prof with $XX million of external funding, you don't have to give a s$%t about anybody, and by now everybody else in the discipline knows that much about you already.