Let me add on to Davidmh's answer aboveDavidmh's answer above. The major issue is that you're not switching from philosophy to mathematics at this stage; you already made the switch a long time ago. As far as the graduate schools are concerned, you will be evaluated as a mathematics major would be.
If you believe your study of philosophy will help make a case for your admission as a mathematics graduate student, then you should mention that in your statement of purpose. However, you need not obsess over it; no one will say that you're unable to pursue graduate studies in math just because you started out in philosophy.