Ignoring him and refusing to respond to any uncomfortable advances or comments might be best.
You don't owe him anything, especially the satisfaction of making you feel uncomfortable. Focus on your poster and on interacting with those who are interested in it. Do what you should do anyway and make some professional contacts at the session.
There is no need to do anything more than if the next position over was by someone unknown to you and whose work you aren't interested in.
You have just as much right to your space as anyone else. Dominate that space and try to let the rest go. With practice such uncomfortable encounters, that may occur from time to time with others, will become easier to manage.
And by "dominate that space" my intention is beyond the confines of a poster location.
There is an outside chance that the conference leadership would respond positively to a request to be moved. "I'd be more comfortable elsewhere" is all you need for an answer if asked why.