A very important fact to be aware of is that Dunning-Kruger is a a cognitive bias that is about skill, not intelligence or ability/capacity to learn. So for instance a person who is a terrible driver may not know they are so bad because of Dunning-Kruger, but none of this tells us absolutely anything about their ability to improve their driving skills with practice and dedicated effort. The Dunning-Kruger effect on the unskilled makes it harder for a person to become aware of their shortcomings, but it says nothing about what happens when they realize they do in fact lack skill! On the other hand, the effect on the skilled is that they falsely believe that what is easy for them is - or should be - easy for others too.
The Impostor syndrome, by contrast, is about how a person assigns causation/responsibility to their own achievements and abilities. A person with Impostor syndrome doesn't say, "I'm at least above-average", unlike the unskilled person under the effects of the Dunning-Kruger bias (who doesn't necessarily think they are great, but just believes they are above average). Nor does the highly skilled person with Impostor-symptoms say "this is just easy, anyone can do this" - because that would actually be acknowledging they are in fact good at something other than fooling people into thinking they are good at something! No, the Impostor thinks everyone else is wrong in thinking they are good or intelligent or did something great.
So while both poles of the Dunning-Kruger bias and the Impostor syndrome are commonly experienced in academia, they don't actually exist on a single continuum where one is forced to try to balance between them. Therefore you actually have the possibility to live relatively free of both of them.
How do you do that?
Um....I'm not sure that I know?
...well, I suppose the best place to start is taking some time regularly to expand your circle and be around a variety of people. Volunteer in community programs (not merely academic ones!), spend some time on a sport, etc. In short, don't spend 100% of your time with students and professors, because you will get a warped sense of what everyone else in the world is really like and you'll tend to get some really unrealistic self-images going. It's the intellectual equivalent of getting your idea of normal physical appearance from fashion/exercise magazines and television. You don't want to get your perceptions all pretzel-like, basically.
There are also specific things you can do - like if you think you are just an idiot and you are just fooling everyone, read some YouTube comments for awhile; if you don't get the feeling that you are actually smarter and more reasonable than a lot of people in the world, seek immediate professional counseling because that's a real problem.
You can also develop some strategies for "sanity checking". For instance, you have taken previous classes and did well, but I'm guessing some of those classes weren't easy and you could have done better in some. Based upon the available evidence, doing a thesis as you propose probably won't be easy. There is also no evidence that you aren't capable of it at all - otherwise you'd have failed all those other classes. Chances are, at your level of math there are no free points for putting your name on the paper - if you got a score higher than 0, you know an awful lot more than you realize.
So you face a challenge - or a threat - depending on how you look at it. If you are interested and excited and don't feel it's a ridiculous impossibility (like, say, being drafted by the NBA) then you have a potentially interesting challenge. Can you do it? It's a distinct possibility, sure.
So now I'd advise the usual general decision-making strategies - pros and cons, risks and Plan Bs if things aren't working out, etc. In short, what you can try to balance is between the extremes of mindless confidence (Leroy Jenkins!!!!!) and constant impending doom (I am so screwed...). And I think that is pretty much just the challenge of life in a nutshell, so if you can master it before you die (or maybe just within a few thousand lifetimes) you'll have done something pretty darn amazing.