Suppose a person writes a research paper without doing a complete literature survey, and sends it to a peer-reviewed journal. Assume that the results in question have been obtained before by someone else. Is it mandatory for journal editors to check the novelty? If not and if the editors also don't know about the pre-existence of this research, then who is liable for the aftermath?
As I see it, the reviewers and editors don't have an obligation to redo the literature search. However, the reviewer should be someone who is sufficiently familiar with the topic that they would be reasonably likely to know about relevant prior work, and should be able to identify serious omissions in the literature search. A reviewer who doesn't have that level of expertise should not be assigned, and if assigned should probably decline to review the paper.
Of course, if the reviewer does know about prior work that duplicates the paper under review, then they should require the authors to cite it, and evaluate the paper based on its novel contributions only, if any (or as an independent simultaneous discovery, if the timing makes that a possibility).
Final responsibility for the content of the paper always rests with the authors alone. However, if the paper is accepted and later found to be a rediscovery, a common course of action is to print an "acknowledgement of priority", a short note citing the previous paper and acknowledging that it was first. Typically it wouldn't be retracted. (Of course this only applies if the editors believe the paper is really an honest independent rediscovery; if it is plagiarism then it should be retracted.)
I would like to start with your assumption.
It is possible but not very likely that a submitted paper is more or less a duplicate from existing published research. But if it is, a duplication would be a fair reason to advise reject.
It is more likely that authors have missed relevant research. In that case a constructive reviewer will suggest the authors to include that research. However, be prepared that the authors did not miss that specific research but have good reasons for not including it.
Another common situation is that a reviewer judges the work against his/her knowlegde base and is just not able to comprehend a different perspective on a similar problem (confirmation bias). Then the reviewer will easily perceive the work as a (wrong) duplicate while it is not.
Then your specific questions:
If the editor did not perform a desk rejection, the editor perceives the work as potentially publishable. In my view, editors are qualified professionals who assess the interest of the submission for the journal’s audience. Although editors are not specialists in every field, the experienced ones will most likely notice real duplicates.
Who is liable if it goes wrong? The authors have the biggest problem because their reputation is at stake. The editor can decide to withdraw (retract) the published paper (not good for the journal’s reputation). The reviewers stay anonymous. But in my view all stakeholder share responsibility.
There are around 20,000 papers added to Pubmed each week. Even my narrow field adds 230 a week. Being able to know all the relevant literature for a paper is impossible. Thus it is not a question of whether the current work missed anything from the literature review, but how important are the things that are almost certainly missing.
Part of the job of a reviewer is to be familier with a field and to know whether there are any glaring omissions in the review. Anything in any of the interdisciplinary journals or the fields top in-house journals. They might do a quick search to see if anything blindingly obvious jumps out that the happened not to know about. But no one can be certain that something important hasn't been missed.