Being sure and being right
Ask a group of people how confident they are in their answers to trivia questions, and a common finding is that their confidence runs higher than their accuracy. Those who say they are 90 percent sure are often right noticeably less than 90 percent of the time. This is overconfidence, and it appears in many fields, including forecasting, planning and games.
In games, overconfidence can mean overrating your chances of completing a draw, assuming an opponent holds a weaker hand than they do, or believing a plan is more likely to work than past results support. The useful question is not whether you are ever overconfident. It is how far your stated confidence differs from your actual hit rate, and whether you can measure that.
What calibration means
A person is well calibrated if events they call 70 percent likely happen about 70 percent of the time, those they call 90 percent happen about 90 percent of the time, and so on. Calibration is a property of a set of predictions, not of any single one. If you say “20 percent” and the event happens, you were not wrong: 20 percent events occur one time in five.
Being calibrated is different from being right a lot. Someone who can only make easy calls may be right most of the time and still poorly calibrated if they call everything 99 percent. Likewise someone facing very uncertain situations can be well calibrated while often being wrong, because they have said honestly that the outcomes were uncertain.
Test it with a prediction log
You can measure your own calibration with a little discipline.
- Make predictions before the result. For each question, write the event and your probability, for example “I will complete my flush draw by the river: 20 percent”, or “the next boss attempt succeeds: 60 percent”.
- Record the outcome. Yes or no, nothing else.
- Group the predictions. After you have a good number of them, sort them into bands such as 0 to 20, 20 to 40, and so on.
- Compare. In each band, how often did the events occur? If your 80 percent band came true half the time, you are overconfident there.
The method has limits. With few predictions in a band, the observed rate is noisy, so a single miss can look like a pattern. The same concern applies to sample size and statistical significance in any game data. Aim for many predictions before drawing conclusions, and treat early results as hints; confidence intervals are the usual tool for saying how uncertain an observed rate is.
A scoring rule: the Brier score
A simple way to put one number on forecast quality is the Brier score. For each prediction, take the difference between your stated probability and the outcome (1 if it happened, 0 if not), square it, and average across predictions. Lower is better.
Take two predictions. You said 80 percent for an event that happened: (0.8 − 1)² = 0.04. You said 60 percent for an event that did not happen: (0.6 − 0)² = 0.36. The average is (0.04 + 0.36) ÷ 2 = 0.20.
For comparison, someone who always says 50 percent scores (0.5)² = 0.25 on every prediction, so 0.25 overall. A Brier score below 0.25 on a mix of events that really are uncertain means your guesses carry some information beyond a shrug, but it needs many predictions before that comparison means much.
Why we tend to be too sure
We remember hits. Right calls are vivid, and misses get filed away, which links to confirmation bias.
We underestimate how much we don’t know. Plans feel more reliable than the typical record of similar plans.
Skill confuses luck. In games with chance, a few good results can be mistaken for skill. A strong run is easily taken as the new normal, when regression to the mean says it will probably be followed by more ordinary results.
Ranges are too narrow. If you give a range you think has a 90 percent chance of containing the true number, it usually contains it less often than that. Widening ranges is a common correction.
Improving calibration
- Consider the opposite. Before committing to a number, list a few reasons you could be wrong.
- Start from base rates. Ask how often similar situations turn out a certain way, then adjust. When the base rate comes from real counts, such as when counting outs in a card game, you can ground the estimate in arithmetic instead of feel.
- Use coarser steps. Saying “about one in three” is often more honest than “37 percent”.
- Check against feedback quickly. Games where you can log many predictions, like a repeated card situation, give you faster practice than ones with rare, long events.
- Avoid 0 and 100 percent unless the event is truly impossible or certain.
A caution
Better calibration helps you reason more clearly about odds; it does not make results predictable or guarantee an advantage. Many games are dominated by chance, and a well-calibrated player can still lose often. What you gain is an honest sense of how much to trust your own estimates, and that is valuable whatever the game.