Is accepting or rejecting a paper the reviewers’ responsibility? Or is it left to the editor to decide based on all the reviewers’ comments?
Reviewers can only recommend acceptance or rejection (or a major or minor revision). It's the editor's responsibility to weigh the reviews, along with the manuscript, and decide.
After all, you will often have multiple reviewers. And they will usually not even know who else is reviewing a given manuscript. So how else could you decide what to do with a manuscript where two reviewers recommend a major revision, while the third recommends rejection?
Accepting or rejecting a paper is always left to the editor. The reviewers, however, are typically expected to summarize their review by providing a recommendation to the editor for the paper's fate. This is generally not just accept or reject, but may also have higher granularity such as "major revision," "weak accept," "reject but encourage resubmission," or "borderline."
As an editor or program chair, I appreciate this feedback (even if I may sometimes ignore it) because sometimes it is sometimes difficult to judge a referee's judgement from the tone of their review. For example, a very long and harsh-sounding review may actually say something like: "I am being very strict in my review of this paper because I think it is good work that can be a great paper on revision." I might overlook it amidst the harshness... or the reviewer might not be so clear.