There are many different notions of quality of a paper:
- accessibility of the presentation,
- soundness of the arguments and conclusions,
- relevance of the research.
Ideally, the level of journals only differs in Point 3. Realistically, there are some differences in Point 1 as well (ironically, I find that middle-level journal score best here, but that’s a different story). However, even mega journals aspire Point 2 – and outrage ensues if they clearly fail.
I would therefore not let the level of the journal influence assessment of deficiencies with respect to soundness, i.e., if I consider a paper unsound, I recommend to reject it, no matter the level of the journal.
Also, think for a second what would happen if all reviewers would recommend to accept all papers that are better than the worst paper in the journal in question:
Due to the variability and laziness of reviewers (and the abundance of bad papers), the threshold would decrease perpetually.
Nobody wants this.
Also, irrespective of the journal’s level, I note everything I see wrong with a paper, and let the editor decide whether this is a sufficient reason for rejection. As language problems (and some other presentation issues) can be fixed by a copy editor or similar, I would not even let them influence my main recommendation, but just recommend heavy copy-editing or similar to the editor.