I went through the same dilemma during my PhD in engineering and solved itthe same problem by taking some courses for credit and auditing others--Introduction to Wines, Existentialism or Marxism, Latin, and Religion and Reason. In my university we also have to take two minors when getting a PhD, and I Waswas able to squeeze some classes I was interested in in one of my minors--Project Finance and Business Strategy. My adviser wasn'twas not much of a fan of the idea, so I had to compensate for it by working harder.
Something else to keep in mind is your funding situation. In certain departments students have funding almost guaranteed for almost whatever time they need to graduate. If that is your case, why not just take an extra semester or so to graduate and take a few more classes? Otherwise, just try to gauge what your adviser would let you do and how much you would be able to get done. I do think it is worth it sacrificing a bit of the production work forto become a more well-rounded person, even because being a good thinker, good communicator, and more empathetic person, all of which you can improve by taking philosophy and other classes, will definitely help you in your future profession.