tl;dr: Your editor is right: a limited bibliography reflects poor quality of the literature review. Improve your manuscript by finding relevant work of others and comparing your results with them.
Even if your algorithm is completely new — and this is a big if, because most new algorithms are actually variations and combinations of previously known ideas — you probably want to show that it is useful. To demonstrate this, you need to apply this algorithm to solve an important problem, and compare the results of your new algorithm with other methods that were previously used for this class of problems. Hence, you need to discuss the origins of your problem, the context in which these problems appear, what other methods have been proposed, and why are they not good enough, so your algorithm is needed. All these statements must be supported by references.
An overview of the research landscape is an important part of any research paper, and the bibliography supports this part and reflects its quality. If your bibliography only includes the papers of your research group, it is very likely that your research misses some aspects of a bigger picture. In this case your paper can mislead readers in believing that you are the only group actively working in this area. Your editors are right in rejecting it.
So my advice is: add more context about the problem (with citations), and about the work of others (with citations). Compare your algorithm with other methods used for the same problem. Clearly demonstrate in your manuscript why your new algorithm is not just new, but more useful, than previously used methods. Then submit the manuscript again.