I think it's completely appropriate.
As an anecdote, when I was an undergrad I was in a similar situation. The professor of a WW2 history class I was taking invited all 20-or-so of us undergrads to his home for a barbecue and tour of his collection of WW2 era guns, equipment, newspaper clippings, and so on. It occured before final grades came out.
In hindsight, it was one of those great lasting memories you make in school. I ended up majoring in a different field, but have since still kept a keen interest in history because, at least in part, of the experience I had there. Had that professor decided not to have such an event I would absolutely consider my undergraduate experience to be lesser than it was, and I think the same is true for many of my classmates both in that year and every other year the professor held such an event.
We, of course, don't know you or your lecturer, but I would be absolutely shocked if he has anything but pure intentions. If you don't want to go just reply to the invitation that you've got a prior commitment and can't make it.
The chance that your lecturer will be going over final grades, see your name, and think to himself "bah, Jonas didn't come to my party, minus 5" strikes me as vanishingly unlikely.