I have exactly same story!
Its very pity that "serious" people behave like that, like offended and trying to get rid of you. Unfortunately, sometimes we are covered with just bad people. Thus, no reason to spend time on them. I recommend these solution:
- find another advisor
- find third-party solution officially staying with current
- work solo
Bonus tip: always use official communication channels (like university email) with archiving all messages. Never speak about late/early but write and then just mention during meetings. This helps in conflicts and really cools down harassment level from your super.