Recently, I need to take some recommendation letters from our collaborators but I'm not sure is it fine to ask them directly by myself or should I ask my adviser to do so? Any idea or suggestion is appreciated.
It's always better to ask a person directly for the letter of recommendation. That way you can judge the person's enthusiasm and interest in writing a letter. They will also be able to give you more direct information than working through the "middleman" of your advisor.
If you feel uncomfortable approaching your collaborator for a letter, that's probably a sign that you might not get the quality of letter you want.
But you should also mention to your advisor that you are asking your collaborator for a letter, regardless of which path you pursue.