Mathematical cryptography is still a largely academic subject without much direct applicability. Your chances of getting an academic job are better than in pure Mathematics, but far from guaranteed. However, you are getting ahead of yourself, since you are still in the middle of your undergraduate degree.
I would not agree that cryptography is applied number theory, even so number theory and algebraic geometry has provided problems that are intrinsically difficult to solve without some inside knowledge.
Modern cryptography as an academic discipline is mainly based on complexity theory, which is currently in a very unsatisfactory state as the NP = P problem is still not solved. Complexity theory is rarely taught at the undergraduate level and the theory of computing receives less and less attention in undergraduate CS curricula. The main question here is the possibility of unbreakable crypto-systems.
Then there are the algorithm / protocol builders. This endeavor combines a rigor even more stringent than used by practicing mathematicians and you might be able to apply knowledge in pure Mathematics that no one else has yet thought about applying. But a good basic understanding of Computer Architecture, Operating Systems, and Networking is also required. This is very similar to the development of abstract distributed algorithms that now are and have been integrated in many cloud applications. If you are good in Mathematics, you tend to be good in this part of CS.
Finally, there are the cryptanalysts in hidden dens in Virginia and their homologs elsewhere. They presumably know a lot about the previous aspects, but I have no good knowledge about it. Usually though they make progress not by breaking cryptography, but its application, which again indicates the importance of knowledge about computers.
So, to answer your question: Of course it depends, but you should expect some extensive knowledge of CS as necessary for a career in cryptography.
Finally, I would like to challenge your impression of Computer Science. A typical undergraduate degree will first introduce students to coding, and only then introduce the big topics, such as Operating Systems, Computer Architecture, Networking, Programming Languages, and Algorithms. Your experience in these classes will be quite different from a programming class, even though instruction usually assumes that students can program. Some of these topics are quite mathematical (e.g. algorithms.) If you are lucky, you can even take a class in theory of Computing, which is essentially Mathematical.
I would encourage you to follow your heart and study pure mathematics. When you reach the beginning of senior year, you can start make plans for later. But for practical reasons, I would also think that taking classes in computing such as a minor in CS might be good for your future career. After all, a Mathematics degree is quite flexible. Did you know that Math is apparently one of the best preparations for Law School? If you are looking for alternatives, look at data science, which is at the intersection of Business (or a similar application domain), Statistics, and Computer Science.