In the U.S., this idea is fraught with difficulties, but is not impossible, in part because the usual undergraduate math curriculum in the U.S. is pretty thin and slow in any case, so that the typical first year or two (or more) of graduate work in mathematics is still coursework, getting-up-to-speed. Despite rumors of "undergrad research" in the U.S., genuine such is quite unusual, although "research experiences for undergrads" do give a positive experience showing that "classroom math" is not what mathematics is eventually about.
But there are intangibles acquired by doing that undergrad coursework, including "being on the same wavelength" as one's potential cohort, and having practice understanding what the instructors expect. While it is true that some of this is not particularly constructive pure convention, it does affect communication in both directions. If you're missing this experience, this will be an added catch-up project.
Beyond conventions and standards for communication, there is also the potential issue of accidental self-deceit about the degree or depth of one's understanding, if one has not interacted with other people. It's not that the typical U.S. undergrad curriculum is terribly substantial, but in a way this makes it all the worse, insofar as the truly important points can be lost or misinterpreted in a context of vast ocean of seemingly uniform technical details. Or, from another side, a too-physics-y attitude about mathematics may generate lots of trouble for you in a "strict" mathematics context.
After these cautions, I guess the point is that it is nevertheless possible, if one really wants it, to pursue mathematics (e.g, in the U.S.) despite not having the corresponding undergrad degree. Your issues would be primarily two: (1) getting letters of recommendation from mathematicians with PhD's, (2) demonstrating some self-study knowledge/awareness despite lack of transcripts showing such. Prepping for the GRE subject test in math might be feasible, and getting a good number is plausible without the undergrad degree, but this wouldn't be sufficient.
Probably taking at least one, probably two or more, upper-division or intro grad level math classes at a serious university would help you generate letters and also demonstrate that you can do the work at that level. At many places in the U.S., it is possible to register for such courses as "non-degree student", paying a lower tuition.
