It is not so much a difference in skills, but the philosophical approach that determines their use.
A physicist would primarily be interested in questions about what reality is and how we perceive it.
A mathematician would primarily be interested in how we reason about things: in abstraction and logic, in linguistic and non-linguistic mental constructs.
This difference in philosophy changes the kinds of questions one asks and the kinds of answers that are considered "satisfactory".
It also leads to differences in the aspects of the skills (be they computational, relational, spatial or some other) that one hones even when the basic skills are from the same set.
In summary, the difference is not so much in the skill sets, but in the value/onus placed by the "owner" of these skills on them. As a consequence, their use can be different.
Update: Looking through the answers, I realised that Andrew has given a somewhat similar, but more detailed answer. However, he has written it as a physicist, and I have written it as a mathematician.