I think it's perfectly reasonable to primarily review changes in a new version of a manuscript that you have already reviewed. I would assume that most reviewers do this rather than pretending to start from scratch, though it is likely necessary to go a bit beyond "just by reading the point-by-point response" to accomplish this.
Consider all changes
I would say a reviewer should consider all changes, not only changes made in response to their own comments. It's possible you will find fault with the way the authors have responded to some other reviewer's comments, or the authors may have made other unrequested changes as part of their own iteration, as indirect response to a reviewer comment, or in response to other new information in the field.
Use a complete "diff"/tracked changes, not just author responses
If the authors/journal have not provided a "tracked changes" or "diff" version of the manuscript, I'd recommend making your own using software to verify that the authors didn't make any sneaky other changes.
Cross-reference changes throughout the manuscript
For a thorough review it may be necessary to cross-reference changes listed with other sections of the paper. For example, if you suggested a change in methodology and the authors made changes to their results section, you should be checking that this methodology is properly described in their methods section as well as elsewhere in the paper.
One example I've often found in published papers are cases where I suspect a reviewer had the authors correct some statistical mistake like concluding two groups are "the same" or "not different" because a p-value was larger than 0.05. The paper may show the correction in interpretation in the results section, but their abstract, discussion, etc still use the faulty interpretation - a careful review should be able to identify inconsistencies like this.
Not every "major revision" has the same scope
It's not clear simply by the designation "major revision" just how much of a paper is impacted: it's both possible for a major revision to involve practically the entire manuscript as it is to involve a narrow but crucial aspect. A reviewer will have to judge on a case-by-case basis how much of the manuscript they need to check; it's not possible to answer the title question here in a yes/no way that applies equally to every possible circumstance.