A journal drop-down menu for selecting a person's title has four options but none of them are Mr.
(I definitely scrolled down enough.)
As a male, non-PhD holder should I select Mx., or is this a mistake by the journal?
Mx is a gender-neutral honorific. It's probably not a typo.
Dr would typically be used if you have a doctoral degree, and Prof if you are employed in a professor-like capacity (it means different things in different countries, but generally any semi-permanent faculty job is reasonably included). Otherwise, you can decide which of Ms and Mx fits you better.
If you feel that there isn't an option that's appropriate for you, you can take it up with the journal.
Since no one else answered your actual questions:
Is Mx. the default title for [m]ale non-[P]h[D] holders?
it is not. It's a recent gender-neutral form of the usual English titles Mr/Mrs/Miss/Ms which might be allowed for those not wishing to signal or accept a binary gender. It's not standard, and Mr. remains much more common. More common still would be for an untitled person to simply be untitled.
This use seems bizarre to the point of inviting a sex-discrimination lawsuit. They require that you provide a title, with lower-level academics forced to declare themselves either female or other. The only rationale for such a setup besides trolling (cismen who complain have their applications filed circularly) would be if they intend to assign extra credit to applications by self-declared women.
As a male... non-PhD holder should I select Mx... or is this a mistake by the journal?
Yes, you should select it if you plan to continue your application, despite knowing your gender will probably hurt your chances.
Based on the context, the options most likely are:
So you would fall either under Ms or Mx depending on your level of education.
I personally understand it like this:
I do not necessarily see it as a flaw per se as it seems logical to divide per earned title. Just a bit unclear spec..
It is flawed by the idea of having Ms understood as Miss instead of M-s, but other than this it is completely logical!
Tell the official to clear things out by giving a hint.