This may depend on the conference. You can contact the program or conference chair for guidance. However, I suspect that the answer will be yes. They want high quality papers presented in the conference and in the proceedings, which is why you get reviewer feedback not just an acceptance/rejection decision.
But don't expand the length of the paper too much or change its structure more than necessary. If you do, it may need additional review, and then you run up against conference deadlines which might result in rejection.
But it is the committee that decides these things, not consensus here. Ask.
And it is probably best not to think of the changes as a "rebuttal". Disputing the referees is a bad choice to include in a paper, though I realize you aren't suggesting that. Revisions is a better way to think of it than rebuttal.
But note that the referees are offering advice, not instructions. You are free to accept or reject that advice. But you should neither just ignore it, nor dispute it (in the paper). Just use the advice to improve the paper as you see fit.