I did my undergraduate in math and physics with equal coursework from both. I am now a PhD student in a physics department with an advisor who is a faculty member in both the math and physics departments. I am starting my fourth year as a graduate student and I've been working under my advisor since the end of my first year. The first year we did traditional physics. I got one small paper out of that work, but at some point I realized that I'd be better off in a math department. I spoke with my advisor about this, and after telling him that I have a bachelors degree in math he was happy to put me on another project that is in probability theory. It has some vague connection to physics, but we are only doing math and I really enjoy the work so far.
I'm firstly concerned about my foundation in math. Other than a graduate course I took during my time as an undergraduate and two other courses I took from the math department of my current institution I am largely self taught. For example, I have taught myself measure theory, topology, functional analysis, probability, PDE, complex analysis, a bit of harmonic analysis...etc. I sometimes worry that I do not have as good of a foundation as a traditional math student who took courses and had to pass a qualifying exam. I've had no one "check" my math in the sense that I have not had to take any qualifying exam or do any graded homework problems. Based on the qualifying exams I've seen from other schools, I think I would struggle a lot if I had to take one of them right now without any preparation. In some sense I feel like I am a mathematician without any qualifications (for example, getting accepted in a math department or passing courses/a qualifying exam), no different than an enthusiast who studies graduate textbooks in their free time.
Secondly, I am concerned how this will affect my academic career beyond my PhD. I would like to become a professor in mathematics. Considering I have self-taught myself almost all the graduate math that I know, will I be taken seriously coming from a school whose physics department is not nearly as good as the math department? Due to my unusual circumstance I am having a hard time networking as most math graduate students/faculty in my school write me off since I come from a different department. I am also having a hard time within my own department as my work is not similar to anyone other faculty/student in physics. As a result I feel academically isolated.
I am worried that I've made the change too late. I've been studying intensely the last year and I feel a little behind; I think I am on par with a second year math graduate student but not a fourth year math graduate student. Is it worth chasing a career in academia as a mathematician or is it unrealistic given my situation?
I've already spoken to my advisor about my situation but I would like to hear some outside opinions as well.