I am now doing my Ph.D. in Europe and I think I am not a good fit for my research group. It is not possible to change to other research groups in Europe and I was planning to drop out of my program and then reapply to some other group working on a different topic. I need some advice on whether it is possible to do so?
Context: I have had candidates in similar situations apply to positions I offered in the past.
It's certainly possible - you are not a slave to your current group, and there is no law that says that we cannot hire people who have already started (and quit) a PhD elsewhere.
However, in practice, applicants in your situation raise an interesting question for the hiring person. On the one hand, you already have some experience in research, potentially decreasing the amount of training required before you will become effective. On the other hand, people will be wondering what went wrong the first time, and whether the same is likely to happen in your next PhD position.
In truth, most such candidates end up not being accepted. There is normally at least one other highly qualified candidate who is perceived as lower-risk. However, oftentimes this is also because the candidates don't do a particularly good job explaining their situation, and transparency (and plausibility of the story) are key. The instinct of many candidates seems to be to try to keep the topic out of conversation as much as possible, and only provide highly generic answers when probed. This is not a good strategy, as it keeps people wondering - and wondering people make up their own explanations (or they simply decide that the risk is too high).
I feel that before you go back onto the PhD student job market, you may want to head over to the Workplace.SE and carefully study the (many) questions there related to how to, and how not to, talk negatively about previous employment. Your challenge will be to convince your next advisor that you are quitting your previous position for good reasons which are unlikely to repeat in the position you are applying to, without thrash-talking your previous research group. It's certainly a capricious edge to walk.
It depends a lot on which university you apply at and what the new supervisor thinks about it. In most cases you would be asked if you have been enrolled in a Ph.D. program before, wether you finished it or not and if not, why you dropped out. In many cases it would be up to the new supervisor to make the decision. It's generally not seen as something positive, but if you are the best candidate, then it shouldn't be a problem.