You are not the first person to think about this idea.
People use similar ideas in forecasting and your idea is similar to the Brier score.
The premise of this foresting is to be the best at predicting the future.
People weight their certainty of predictions over time (e.g., the probability of rain tomorrow is somewhere between 0 and 100%).
Over the short-term, the Brier score penalizes certain wrong answers (e.g., if you say there is a 95% chance of rain but it does not rain, your score is lower than if said there was a 55% chance of rain).
But, over the long term, this method penalizes both uncertainty and inaccuracy (e.g., if I consistently make right predictions with 55% probability, I will have a lower score than if I make correct predictions with a 75% probability).
The Good Judgment Project uses Brier scores to evaluate people's ability to forecast and would be a good starting place for you to you look.
That being said, I don't know how your proposed idea would work on exams unless you wanted your students to study game theory, forecasting, or predictions rather than your subject material.
In general, many if not most education experts think multiple choice questions are a bad way to assess learning (e.g., for some of the reasons described in this article).