I recently graduated with a BA in physics. Towards the tail end of my undergrad career I took the Putnam exam (scored 28 w/o having taken any upper div math), and subsequently took a couple core math courses (one quarter of real analysis, one quarter of proof-based linear algebra). These were the probably the three most enjoyable learning experiences I've had in undergrad, and for the past month or so I've been trying to think about how I could possibly switch to a research career in mathematics but it seems very unfeasible due to the small amount of core classes I took and my inability to do any substantial math research both due to lack of knowledge and lack of faculty connections.
Right now I am taking a gap year working in a physics lab. I am technically able to audit more core math courses at my undergrad institution but wouldn't reasonably be able to do more than two (maximum three) more. I could self-study but am worried about the lack of grades to reflect my learning (although I could take the math GRE and hope for the best). It's my understanding that I'd need a strong foundation in analysis and algebra before being able to do any sort of meaningful research, so my hope is now to get into a master's program where I could fill in my gaps and start working on research, but even these programs require a much more thorough foundation than what I'd be able to build on paper. Is there any way that I could switch to math at this point, with the goal of entering a math PhD program, or is my best bet to try to do theoretical physics and segway that into mathematical physics?
Thanks in advance for your time and answers.