(I am a tenured professor in the math department at UGA who was on the committee that did graduate admissions for four years.)
If we have to make a choice, then math graduate programs definitely care more about your math GPA than your undergraduate GPA. We also care, equally, that you take the most challenging and graduate-preparatory math courses that can. So a math GPA which is close to 4.0 including year long courses in analysis and algebra, some exposure to geometry and/or topology and at least one graduate course would make you a strong candidate for most graduate programs in mathematics. Certainly one or two subpar grades in other courses are easy to ignore.
Having said that, it would still be better if you got good grades in all your courses, and admissions programs do care about that to an extent. [Added: Wait, I didn't say exactly what I meant here. Not good grades in all your courses, but good grades in most of your courses, as reflected by a good GPA. We will almost never look at individual course grades outside of mathematics and very closely related fields.] Here are some reasons:
1) If you have across the board difficulty with courses in the humanities and social sciences, then that signifies either a lack of linguistic or writing skills or an inability to focus on a single topic (generally you spend longer writing a paper than doing a single problem set, and the paper itself is one big task whereas the problem set is a collection of largely independent smaller tasks). But skills in language and writing and the ability to focus on a single topic both play a larger role in the study of mathematics at the graduate level (and beyond) than the undergraduate level. This kind of poor performance would be exacerbated by a subpar GRE verbal score, and conversely a high GRE verbal score would partially offset it.
2) Some academic scholarships take your overall GPA into account. At my university eligibility for the highest class of internal graduate scholarship is determined by the student's "academic index", which is calculated using their GRE scores and overall GPA. This may be a more extreme situation than the norm, but in my experience many if not most scholarship winners are students of the bulletproof credentials variety.
3) In line with the above, you have to remember that the graduate admissions process is highly competitive. Most math PhD programs in the US only admit students who are fully funded, and therefore by no means could we admit all fully qualified applicants. (Even without funding it is clear that finiteness conditions on the faculty, the course offerings, the facilities and so forth must impose an upper limit.) So if for instance you have a 3.8 math GPA and 3.0 general GPA, you will almost certainly be ranked ahead of candidates with a 3.0 math GPA and a 3.8 general GPA (assuming other factors, mainly the difficulty of courses and the quality of recommendation letters) are equal. But you will be ranked behind some students with a 3.8 math GPA and a 3.8 general GPA. Such students exist, and it only makes sense to take this information into account.
My best advice to you is to realize that your overall GPA counts for something and therefore do whatever you can to get strong grades in all your courses without sacrificing your efforts in your mathematics courses. Something that you might want to think about: why are you not doing as well in your non-math courses as you are in your math courses? I can't think of an answer that wouldn't point the way to room for improvement. Some possible answers:
- English is not your first language, and your reading / writing skills place you at a serious disadvantage in certain courses.
This is a very particular situation. If you're in it, you should take pains to demonstrate that your English skills are good enough. Probably you should take and do well on the TOEFL or similar exams. You should also mention in your personal statement that English is not your first language but have the English in your personal statement be flawless. The point is that if you're a foreigner and you're English is okay but not as good as the natives, then as time passes it will get better and be less of an issue. Also a C grade in, say, a Shakespeare class by a non-native speaker has a totally different meaning from the same grade by a native speaker. The admissions panel will cut you some slack for that provided they believe that your English skills are good enough for you to be successful in the program.
- You think it is not possible to give the attention you need to your math coursework as well as your general education requirements.
It certainly is possible. The students who are winning the Putnam, taking graduate courses in their sophomore year, publishing solo papers in serious journals...are usually also excelling at all their courses. My own view, which I think is probably shared by many math admissions committee members, is that advanced math courses are much harder than general education requirements. If you read the assigned materials, can think and write clearly, and give yourself enough time to write suitably and according to the requirements of the assignment, you'll get good grades in these courses. It does not even necessarily take more time to get an A- in a humanities class than a B-: it's more a matter of having your act together. (In math courses I have seen students stretch themselves to their limit to get a C and actually been quite proud of their achievement.)
- You haven't mastered the skills necessary to succeed in writing-intensive courses.
As above: you should work on this before you get to grad school.
- You don't take these courses seriously and didn't realize that graduate schools do.
Well, we do. Most academics take all academics seriously: generically speaking we are "overachiever types" across the board. If you look through the profiles of award-winning undergraduates, you will usually see that they graduated summa cum laude, at the top of their class, as a student marshall...whatever is the local form of highest honors. Success in one academic discipline is undeniably positively correlated with success in another academic discipline. This is far from logical necessity, but it's out there and we do take it into account.
Added: Since I wrote this answer, I became the Graduate Coordinator for the math department at UGA. So this topic is much more vivid in my mind now. Here's what I can report: most applicants have very good GPAs. The median undergraduate GPA (all classes) of all our applicants this year is around 3.7. I dug a little deeper, and among American students there is a high correlation between undergraduate GPA and other metrics (including various GRE scores). Let me also say that we looked specifically for candidates with very low GPAs but did not reject any for this reason alone. Interestingly, many of the (not many) applicants with very low undergraduate GPAs also had not great undergrad performances in math but had nevertheless gone on to a master's program (not at Princeton...) and put up a much more respectable graduate performance. The brilliant math student who blows off all their other classes is indeed not much seen by us.