When it comes to suggesting changes, it is true that most reviewers will suggest where to add material. As the general purpose of these suggestions for changes is, however, to make the paper "better" (whatever that means in the concrete case), removing information can be fully compliant with that goal.
Now, you list two very different situations:
there are cited references that are not related to the discussed subject
These references do, by their mere presence, reduce the quality of the paper (if we assume a paper should be as concise as possible when it comes to providing information unrelated to the subject of the paper). You cannot truly tell whether such references were added with malicious intentions (for example, to increase someone's citation count) or simply by mistake, so you probably should not make any direct accusations, but a request to remove the respective references is well appropriate.
or there are elements in the bibliography that are not referenced in the manuscript.
This case, on the other hand, depends entirely on the guidelines of the venue you are reviewing for (provided that the references are indeed on topic for the paper):
- If these guidelines require references to all bibliography items within the paper body, go ahead and suggest either their removal or adding a mention in the paper body.
- If, on the other hand, the guidelines do not contain this requirement, keeping these references improves the paper, as it helps readers who browse bibliographies for related work in the subject area by pointing out additional resources.