It may depend on who your advisor knows in foreign countries --- or who you can get to know in foreign countries.
I've known people who've done this, but my impression is that in grad school it becomes less about "doing a study abroad" and more about doing a "visiting scholar" program at a particular school. That is, you wouldn't seek or find something like "I want to study abroad" or "I want to study in Country X". What you could do is find a particular school, department, or researcher in Country X and arrange to visit their department for a certain length of time. I'm in a department in the US, and we regularly have visiting scholars from a range of countries (e.g., Japan, the Netherlands, Finland, Brazil). We currently have one of our own students on a visiting scholar stay in Germany and another is planning a possible study in Singapore.
The thing is that, like anything else in a PhD program, "studying abroad" can't just be "studying", it has to be connected to your particular research. So you would need to build connections with particular departments.
Also, there can often be opportunities for shorter-term study abroad in the form of "workshops" or the like. These would be more in the range of 2-6 weeks. That's a lot less time abroad, but my impression is it's much more straightforward, because you just apply (instead of having to form specific personal relationships with other researchers). These workshops sometimes have fellowships available to cover some or all of the cost, and even if they don't, it's likely to be easier to get funding from other sources if you can point to a specific "thing" you're using it for (i.e., "I am going to this workshop on these dates to learn about these topics" rather than "I'm going to Country X for a while to hang out").
