This is perhaps just stating the obvious, but whether or not your paper gets published isn't up to the referee. It's up to the editor. So if you received a generally positive referee's report then you can just tell the editor that the requested revisions are outside the intended scope of the paper and decline to make them. The only way I could see this becoming an issue was if the referee's report made it seem as though your paper wasn't publishable without the requested revisions.
Added (as an extended comment to the OP in light of their comment to the above): You seem to have a very defeatist outlook about the publishing process. For starters, the referee might not be all that much more senior than you are. I referee papers all the time (and have done so since my first year post-PhD) and I won't be going up for tenure for another couple of years, for example.
Regardless, all that the referee has done is write a referee's report which ideally makes some sort of recommendation to the editor. But that's all it is. A recommendation. If you send a response to the editor explaining that you don't think that one of the changes requested is necessary, you shouldn't assume that the editor will dismiss it out of hand. They might, of course, but then again they might not. I've had maybe ten papers where I explained to the editor that I did not agree with one of the referee's comments and didn't make the requested change. Not once did this have an impact on my paper getting accepted.
Like I said in my answer, it could very well be the case that the referee doesn't think your paper is worth publishing without the requested revisions. In this case the editor might side with the referee and you should probably just make the change if you can do so in a reasonable amount of time. But if you give a thoughtful explanation why the referee's recommendation is not necessary, the editor could very well take your side. Similarly, if the editor asks the referee what they think of your comment, they might decide that while the paper would be stronger with their requested revision, it should be published even without it.
Bottom line: It's your paper. If you have an opinion about what should be included in it and what should not be, you should let the editor know and assume that they'll at least hear you out.