I suspect this varies a lot by field. In philosophy it does matter where one publishes; a 2013 poll on a philosophy blog gave the top 7 English-language publishers as:
- Oxford University Press
- Cambridge University Press
- Harvard University Press
- Routledge (Taylor & Francis)
- Princeton University Press
- MIT Press
- Wiley-Blackwell
In my subfield, due to the closer connections with mathematics and computer science, a lot of good volumes come out with Springer, which is nowhere near the top of this list. So even knowing the field is not sufficient to be able to determine what the best publishers are.
As with most questions of this sort, you are probably best off consulting senior members of your field, and seeing where the stars of your field publish.