Ask yourself this: If people knew you stayed on as a Ph.D. candidate because somehow you got to worry more about how this would reflect on your advisor, this in itself would reflect very poorly on him; and perhaps on you as well. So... don't let the how-it-would-reflect-on-him concern worry you.
H-O-W-E-V-E-R! There is the "Neighbor's grass is always greener" effect. It may be the case that switchingleaving your Ph.D. track has detriments you might not be taking into account, and on the other hand, maybe you're stuck in your Ph.D. work and, arranged differently, it might be more interesting and rewarding. I'm not saying that's necessarily the case but sometime it is. Also, some people switch to working on the rest of their Ph.D.s part-time. Again, that doesn't work well for everyone, I'm just saying you need to carefully weigh your options etc. etc.