When applying for undergraduate schools it was relatively easy to look up average ACT/SAT scores and GPAs of admittance to know if I would be a competitive applicant. As of right now I want to pursue a PhD in math and I'm struggling to know if a program is within my reach or not. Last year my grades tanked and I went from a 4.0 math GPA to now a 3.2 with a 3.5 overall. I'm working now to hopefully get that up and I'm planning on taking a year off after where I may stay in my undergraduate town to continue to take more classes to raise my GPA but either way I'll have a C and a BC on my transcript so I don't know which schools are within my reach if I get enough research experience/letters of recommendation and which I shouldn't even bother with.
If you can show real research potential (papers in one of the journals for undergraduate research, participation in research as an undergraduate) that will probably weigh more than raw GPA. A dip in grades is certainly worrisome, as it hints at losing interest. You'll have to have an explanation.
The best type of person to answer this question is a mathematician who knows you and knows roughly what a successful applicant to your desired programs looks like. A bad grade or two does not necessarily imply a gap in your knowledge. Even if it did, it may or may not be a gap that the admissions committee cares about.
Here is one relevant anecdote: I once applied to a top 20 school (math Ph.D.) and was not accepted. I asked the admissions director what I might do to improve my application. He cited some low grades that were a red flag for him: a B+ in linear algebra and B- in abstract algebra. He did not even seem to care that I had a C (lower grade) in differential equations, and he said nothing of my GPA.
To go with my point in the first paragraph, my advisor had in fact (for various reasons) told me not to apply to this school!
It's also useful to identify professors in the department of your target school performing research in which you're interested. When you apply, be sure to mention that professor (or those professors, if their stated research topics are very similar) in your application.
Therefore, to help with your justification of the bad grades, it may be possible to show your good grades in the courses relevant to your research interests and your poor grades in courses irrelevant to your interests. This may also help you avoid worries of waning interest from the admissions committee ("I got so interested in [this topic with good course grades and directly tied to Professor Jones' research area] that I didn't devote enough time to [this topic that's irrelevant to stated research goals with bad course grades], and that subsequently resulted in my low marks in this single course").
It's an idea, but I think anything you can do to make those grades seem uncharacteristic of your general performance and interests will help your case. It's not so much a matter of overall GPA, but how you performed in the courses that count.