I did precisely what you're discussing (attend a semester program at a mathematics institute immediately after graduating), and I'd strongly encourage it.
You say you're worried about wasting six months of time because you don't have a specific problem to work on with the people there. Unless the only work you're expecting to do in the next six months depends on someone who's at your postdoc institution, presumably you can expect to be at least as productive at the institute as at your postdoc.
But as a newly minted PhD, you want to be productive, but not just by continuing to work in the exact speciality you carve out in a thesis. So you don't want to only be working on the specific problems you're already thinking about right now: you exactly want to encounter other perspectives and other problems and get some work done on those.
Which is basically the point of a thematic program. Even if you aren't going in with a specific problem in mind, you may well discover one. Someone may want to work on something with you, or may mention something as an open problem, or it may just come up while talking to people. And even if it doesn't, and you end up doing most of your work by yourself in your office, you'll still get a lot of exposure to the broader state of your field.
Also, of course, you'll meet a lot of people, which may pay off down the road. Even if you don't end up working on something with the person in the next office while you're there, you'll hear a lot about they're specialty, and perhaps a year from now suddenly realize that they're the perfect person to ask about something.
