I'm changing my major to philosophy from computer science in the middle of a semester, and I'm doing pretty poorly in one of my math classes. My advisor told me it would be a good idea to drop it since I'm changing my major. Would it have any impact in grad school admissions, especially if I want to go to a highly reputable school?
In general, no single thing like this would have much impact. In this case, I'd say even less. People aren't looking for small reasons to reject your application, but for evidence that you will be a success in your chosen field as well as in the program you apply to.
So, if you do well overall and do especially well in Philosophy then you should be fine. Good letters of recommendation will overcome a simple thing like a W.
But be prepared, if asked, to talk about why you wanted to switch. And make the reasons positive, about philosophy, not negative about math or CS.
A "W" on your transcript typically comes with no explanation. A pessimist reading your application might conclude that you withdrew from the math course because you were failing the course. In comparison, a B or even a C in the course would look better.
Whether this would be a significant factor in an admissions decision depends a lot on the person reviewing your application.