There are students with math major (and other major) go to physics graduate school, but I think all of them have taken physics courses. The first question you should ask yourself is that why you are interested in theoretical physics. If you were the admission committee and see an applicant said that they are interested in physics but taken no physics courses, what do you think? You must explain it in somewhere in your personal statement.
The first thing you should do is to take physics courses and get good grade on all of them. Also, you should try taking few graduate courses. There are no need to take courses on all topics, you should only have strong interest in few topics such as QM, EM, SM, QFT and string theory. Some of them are very mathematical (that good for you). Doing so can demonstrate your interest in physics.
Second, you might try to apply for master program in physics. It is easier to get in and after 1 or 2 years you can apply for PhD in physics to continue your study.
One more options is to apply for math department, in particular, you should look for applied math. There are usually few professors studying mathematical physics. It is particular true for some countries that the theoretical physicists stay in the math department and experimental physicists stay in physics department. Assuming you have good background in mathematics, it should be the easiest option for you to get in those school and start studying theoretical physics.