I am interested in areas of theoretical physics/mathematics which simply don't exist in my institute. I tried 2 or 3 different groups here which were not in my interest (but I felt I had some transferable skills) and it didn't work out.
There is no age limit for graduate studies. People are free to apply at whatever stage of life they choose, if they feel it's the right move for them.
As an example of this, a very good friend of mine was a social sciences major as an undergraduate, and worked in Washington, D.C., for a number of years before leaving politics and starting a PhD program in medical physics—and he did this in his mid-thirties. I've worked in the same department as other postdocs who made the career choice even later—they were in their early fifties!
So I would not look at your case as hopeless at all. If you find something else that inspires you, go for it.
As your question stands, the only honest answer is it depends but probably yes, it does.
Your question is lacking in essential information for a definite answer. Are you currently doing a masters or a phd? What is your background? Do you want to make a career in academia, ie. research oriented, or in the industry? Are you self-motivated? etc...
I would not rule out changing grad schools based on your age alone as there's no age limit to right a wrong. However, use common sense: be pragmatic and take time to analyze your options. Perhaps your advisor would allow you to collaborate with groups from other universities? Or maybe you could even do an exchange program? Is wrapping up what you have and moving on to something you're more interested in an option? If your project has been given a grant, ponder carefully the implications of leaving your current grad school.
If you feel that changing grad schools is your only option, then unless what is hidden behind starting again is that 400 lb gorilla, I see no obvious reason why you should not live a more fulfilling academic life.
Final note: if I were to review your application to a new grad school, I'd like to understand the reasons that brought you to your current grad school if no group was seemingly doing something that you're interested in.