Let me give an answer from my own viewpoint - that of someone who hires and 'supervises' postdocs on a regular basis (in my field, mathematics).
I consider that a postdoc coming to my group comes for two reasons: (a) to bring new knowledge to our group, and (b) to learn things from our group. Both are good for the postdoc and good for the group - everybody profits from this arrangement. Learning things from our group is best done in close collaboration with the supervisor (me), while contributing to the group often benefits from a much wider scope.
So I tell my postdocs that I expect them to work half of their time on the project they are formally appointed for, and the other half on something of their own choosing - and I suggest that for this part they might also look for other connections within our group or department.
What this means for the OP is (as has already been said): discuss it with your supervisor, and ask yourself the question whether the 'outside' collaboration might also have benefits for your supervisor.