If you haven't taken any pure math courses, you are very unlikely to be accepted into graduate school in pure mathematics. How can you prove to the school that you know the stuff they expect incoming graduate students to know? (The first courses in real analysis and abstract algebra, at an absolute minimum.)

If you have taken pure math courses, then you will have grades in these pure math courses, and these will be taken into account by the admissions committees.

If you have A's in your pure math courses and B's in your physics courses, the likely the admissions committee would weight the grades in your math courses more heavily.

If you haven't taken any pure math courses, you are very unlikely to be accepted into graduate school in pure mathematics. How can you prove to the school that you know the stuff they expect incoming graduate students to know? (The first courses in real analysis and abstract algebra, at an absolute minimum.)

If you have taken pure math courses, then you will have grades in these pure math courses, and these will be taken into account by the admissions committees.

If you have A's in your pure math courses and B's in your physics courses, then likely the admissions committee would weight the grades in your math courses more heavily.