You are in a tough spot and I don't think that any course of action is ideal or without risk. If you can accept this "messiness" and commit to a reasonable course of action, you'll probably come out OK. It's also important to understand that you can't control or determine his reactions, and you can't be responsible for his feelings, as long as you act responsibly and ethically.
Call a meeting, set the agenda
First, I suggest that you schedule a meeting with him to discuss your working relationship. If it were me, I'd speak plainly and directly, without any tone or implication of blame or distress. I suggest that you make these two points:
"I just want to confirm that our relationship is about work only. I am completely focused on my research career, and that is the basis for our relationship. I'm not here for friendship or anything else."
"I'd like to make changes in specific behavior patterns. In these requests, I may be different than other PhD students, but I'm clear about what I need and what I'm comfortable with." Then list the behaviors you'd like to see changed (e.g. no "pop-in" meetings with non-work discussions).
Having a successful meeting
Schedule this meeting. Don't improvise. Don't combine it with any other meeting or topic. Allow for enough time (an hour is sufficient, though you may only need 5 minutes. By scheduling an hour, you are avoiding time pressure for everyone involved) without pressure to complete sooner or worries about starting on time. Plan what you are going to say ahead of time, and even rehearse it, by yourself or with a friend. If you think it's necessary, have an ally with you at the meeting -- a woman friend, anyone you respect in any role in the University, or even someone from the outside. Just say, "This may be a difficult conversation, and having X here to support us gives me more confidence."
Also, if you can imagine that you are asking for something innocuous -- e.g. changing the seating arrangement at a seminar, adding vegetarian options at a department meeting, etc. -- it will help you and it will help him immensely.
You don't need to ask him how he feels about you, to discuss the past or what he was or was not thinking or intending at the past, or any of that. You also don't need to explain how you feel (even though it would be justified). Talking about your feelings in this setting is almost never effective to change behavior.
You don't need to ask him if he's "OK" with your requests, or what he might want as an alternative. There are no alternatives. What he wants beyond your requests is irrelevant.
Also, if he brings up any other subject, no matter how related or how reasonable, you say: "I'm not here to discuss that."
If you haven't picked up on this already: you need to be the dominant person in this interaction. Not flamboyantly or even demonstrably. Just set the agenda, run the conversation, and lead to the conclusion.
Yes, you will be nervous. Yes, you might feel uncomfortable. Yes, you might be seeking his approval and affirmation during this meeting. Let go of all of that. All that matters is that you have this conversation -- short, to-the-point, and direct -- and get to the conclusion you are aiming for.
Finally, if he wants to talk about his needs or experience in the relationship, do that in a separate meeting. Be firm.
Be prepared to set boundaries
Regardless how the meeting goes, it's likely that he'll continue some or all of the behaviors, if only out of habit or faint hope. For each setting and behavior, be prepared to set a boundary -- saying 'no', disengaging, leaving the room, reminding him that you are not 'OK' with this, or what ever you believe will be effective. Not to put him down, but imagine that you are training a dog to not bark or to not jump on visitors. It's just behavioral conditioning.
If you do all this in a way that doesn't publicly embarrass him or privately make him "the Bad Guy", it's unlikely that he will kick you out of the department or university. There's a chance that he might do something bad toward you (many women have experienced negative consequences in similar circumstances), but the odds are lower by taking this path.
Many of the other answers and comments have expressed the view that this direct approach is "incredibly risky" or "likely to backfire" and have suggested more subtle or indirect approaches, including being as "nice" as possible during the process to avoid negative reactions.
My answer reflects my personal and professional values and also my work history (many decades in high tech industry). I'm not naive about power or politics in university departments or research labs.
I believe that it's very valuable and proper for less powerful people to stand up to people in power on issues such as morality, ethics, and even suggestions on fixing problems in the organization (e.g. workload imbalance). 'Standing up" helps the organization as a whole and can be part of a culture change in the organization.
Any professional relationship like this merits a one hour face-to-face meeting if the meeting is about improving the working relationship. Just because the topic might be uncomfortable to one or both doesn't change that. (Such a meeting need not be a "trial" or "attack" as some people have described it.)
One problem I have with indirect/subtle approaches in this setting is that they do not adequately empower OP, implying that she needs to be deferential to her adviser in this matter. I believe that, in the matter of relationship integrity, no one has to be deferential to anyone else. We all have the right (and duty) to stand up for ourselves.
Last, I don't assume that the adviser is a harasser or that he is doing anything that he considers inappropriate. It all may be very innocent and even well-intentioned on his part. If he is well-intentioned, then he'll probably receive these direct communications positively, shift his behavior, and all will be well.