I must say I find this question quite odd in light of the two undergraduate degrees you have: philosophy and mathematics. They have been linked for thousands of years, and some of the most prominent mathematicians of the past two hundred years were philosophers. And some of the most prominent philosophers were also mathematicians. They have gone hand-in-hand for a really long time.
George Boolos, author of the textbook on mathematical logic, Computability and Logic (I'd be surprised if you hadn't used it during your undergraduate studies), started out with a degree in mathematics from Harvard, did a masters at Oxford in philosophy and mathematics, and particularly the philosophy of mathematics, and went on to do a PhD in philosophy at MIT.
People like Hilary Putnam, one of the most famous philosophers alive, have made major contributions to mathematics even outside of the philosophy of mathematics. Putnam published major works on the Boolean satisfiability problem and Hilbert's tenth problem, for instance.
Another prominent mathematician, Solomon Feferman, is also a philosopher.
While certainly there are not a lot of Fields or Abel winners who are philosophers, there is no doubt that the two fields compliment each other immensely.
More relevant is the fact that there is absolutely no doubt whatsoever that there is no way having an undergraduate degree in philosophy could somehow negatively impact your application. It can only improve it in universities where they understand the links between the fields, and in departments where it doesn't, it's going to have no impact at all given you also have a masters in mathematics.
I must say that I am surprised, however, that someone considering doing a PhD in Mathematics would be so oblivious to the obvious, though.