As you specifically mentioned that the conference is not being held in the recommended hotel:
Typically, conference organizers will ensure that there's transportation to the conference from the conference hotel, especially if the hotel is some distance from the conference. Typically, this is handled via buses, although I did have one situation where they hadn't secured enough buses for the number of attendees, so some attendees has to wait quite some time at the end of the day for the buses to drop off passengers and return. (about an extra hour, as the hotel was a 15-20 minute drive away).
This can in part make up for the expenses, but this can cause its own problems -- if you decide to sleep in and take a leisurely breakfast, rather than attend the 'opening ceremony' ** , and you suddenly find that the '15-20' minute estimate to the conference is driving time, not walking. So you're then rather stuck with the conference's schedule until you can figure out the public transit system, or can find a taxi driver who speaks your language. ***
** As it took almost 4 hrs from the time to plane landed to get to the airport the night before (long lines in customs, and you were in the one where the police inserted people to the front if they had detained them, then waiting for the shuttle that the conference had organized to fill up (as you had missed the expected time slot, so were waiting for the next batch of people), and then the 90 mins to the hotel) ... and you're now ~8hrs earlier than where you had started the day before.
*** Because even after the hotel explains to the taxi driver where you're trying to go, they might not actually have a good idea where they're going to, and then you're stuck trying to show him conference materials you have with you that are in English ... which does no good in a Cyrillic language country.
Another advantage is that conference planners will try to select a hotel that has accommodations for people speaking the language of the conference, or they'll assign some translators to the hotel to help. So if you don't speak the local language, and it's a country that's not well known for having a lot of speakers of your language, the conference hotel might be less problematic.
And as you suggested, there's the social issues in being nearby (can stumble a short way back to your hotel), but there's another reasons -- it's only come up once (and not foreign travel), but I was at a conference when the whole city was shut down. (it was Baltimore, and a hurricane ... I think Sandy). A few of the attendees were staying at alternate hotels that weren't within a couple of blocks walk, and as the public transit & taxis had shut down, they had no way to get to their hotels.
Personally, I rarely book the conference hotels (because they're rarely within the allowed per-diem, and I hate filing the extra paperwork for going over) ... but for international travel, if it's in a place that I don't speak the language, I make the exception. Although I will recommend that you get a taxi ride in Kiev at rush hour if you're ever there ... it was like lane markings were just suggestions for inferior drivers.