I don't really get your dilemma unless there are further geographical constraints you haven't mentioned. In the US at least, the quality of the university is not much correlated with the population size or density. The American university currently rated #1 (according to a famous ranking that should not be taken too seriously; but certainly it is and has been a very top university) is Princeton. Princeton is located in such a small town that I did not want to go there for graduate school. On the same list we find places like Dartmouth, Cornell...
I am not nearly as familiar with universities outside of the US, but for instance I was a visitor at Universite de Bordeaux I. It turns out that there is some mildly false advertising here: UNBI is not actually in Bordeaux (which I thought was a charming, beautiful small city, by the way), just very close to it in a tiny town called Talence.
Also many people -- probably most -- who like to live outside of cities nevertheless commute into them for their work. One hour's drive from a prestigious university puts you in almost any kind of living situation I can think of.
Are you really asking whether moving from one of the top ranked European universities to "a Phd position in a very small University in a medium sized Town, in order to relocate and live in a rural village near the town. That university would be quite small and barely large enough to offer Phd programs, and definitely do not have much international reputation or anything. The exact opposite to the university where I got a Master's, which is high ranked and has amazing international reputation" will cause your academic career to suffer? Okay, fair question but an easy one: yes.
What I'd be doing would be the exact opposite of the mainstream, where people are always more willing to live in cities despite the disadvantages,
That is obviously false. Lots of people do not live in cities. In fact, lots of people who would prefer to live in large cities instead take jobs in small cities / college towns in order to keep their academic career alive. (I am one of them. I grew up in a city of two million people. I now live in a city of one hundred thousand people and have a really good academic job there.)
I suggest that you think more carefully about your long term goals. It seems to be implicit that you want to stay in a very small geographic space. That kind of priority is anathema to an academic career, and that's true in Europe as much as or more than in the United States. I know that because I know many excellent European mathematicians who have tried for years to find a job in their home country, to no avail. In many cases they need to leave Europe entirely in order to find a suitable academic job. Or maybe you don't really want an intense academic career following your PhD but would be content to have a mostly teaching position in a desirable geographic location (maybe even at a good high school?). In that case, simply getting your PhD anywhere might be enough. You should look into it. But considering a much worse PhD program wondering how much you're sacrificing by doing so is a very poor investment in your own future.