It is unlikely in the US, but if your acceptance into the doctoral program was specifically contingent on finishing the masters, then, yes, you need to inform them.
But, as a masters is seldom required to begin doctoral studies in the US, I think that scenario is unlikely. You should be able to determine that from the information you were given in the deferment documents.
I'd guess you are fine. You still need to complete all doctoral program requirements, including the (likelYlikely) qualifying exams and the coursework that may be needed to do so.
Good luck in your studies.