The core problem
You switch to a new strategy and win 6 of your next 10 games. Is it better? Probably nobody should conclude that, but why not? Because a player with no real edge wins 6 or more out of 10 fairly often. Statistical significance is a formal way of asking how surprising a result would be if nothing real were going on.
The null hypothesis and the p-value
The starting assumption is called the null hypothesis: there is no true difference, and results are due to chance. In the example above, it says your true win chance is 50 percent.
The p-value is the probability of seeing a result at least as extreme as yours if that assumption were true. A small p-value, conventionally below 0.05, means the result would be unusual under pure chance. It does not give the probability that the strategy works, and it does not measure the size of the effect.
A worked example
Compare 60 wins in 100 games against a true rate of 50 percent. The standard error under the null is √(0.5 × 0.5 / 100) = 0.05. The observed rate of 0.60 is 0.10 above 0.50, which is 2 standard errors away. A two-sided test gives a p-value of about 0.046, just under the conventional line.
Now take 6 wins in 10 games. The standard error is √(0.25 / 10), about 0.158, and the gap of 0.10 is only about 0.63 standard errors. That is entirely ordinary, with a p-value far above 0.05.
This is why short-run results can look dramatic and still mean little. The same observed win rate of 60 percent is borderline at 100 games and meaningless at 10. This is the role of sample size: it determines how much a given gap says.
How many games are enough
The answer depends on the size of the difference you want to detect. Smaller effects need many more games. A rough planning calculation for win rates measured against a 50 percent baseline, using 5 percent significance and 80 percent power (an 80 percent chance of detecting a real effect of that size):
- To detect a true win rate of 55 percent, you need roughly 780 games.
- To detect a true win rate of 60 percent, you need roughly 200 games.
These are approximations, and they assume independent games with a consistent win chance. The second figure reflects the pattern that a gap twice as large needs about a quarter as many games. If an effect is small, a personal log may never contain enough games to confirm it, and the honest conclusion is “can’t tell.”
Power and false negatives
A non-significant result is not proof of no effect. With too few games, a real edge can easily go undetected. Power is the chance that a study of a given size would catch an effect that truly exists. Planning the number of games before starting guards against both over-reading noise and writing off real improvements.
Significant is not the same as large
With enough games, even a tiny difference becomes statistically significant. A 50.5 percent versus 50 percent gap might be detectable in a huge online dataset yet be irrelevant to play. Always look at the size of the difference and its confidence interval as well as the p-value.
Ways the numbers mislead
- Multiple comparisons. Test 20 independent things at the 0.05 level and the chance of at least one false alarm is 1 − 0.95^20, about 64 percent. If you slice your data by opponent, map, time of day and deck, some “findings” will be luck.
- Stopping early. Checking after every game and stopping the moment the p-value dips below 0.05 greatly raises the false positive rate. Fix the sample size first.
- Choosing the question after seeing the data. A pattern found by searching is a hypothesis to test on new games, not a result.
- Non-independence. Streaks from tilt, learning, or the same opponent repeated can break the assumptions behind the standard formulas.
- Regression to the mean. An extreme result tends to move closer to average on retest, which is why a top-ranked performance often looks weaker the next season.
A sensible workflow
- Write down the question and the difference you care about.
- Choose a sample size, using a power calculation or a rough rule like the one above.
- Collect results under consistent conditions.
- Analyze once, at the planned size, and report the estimate with its interval.
- If the result matters, repeat on fresh games.
Sample size also matters for physical randomness. A chi-square test on a die from 60 rolls can only detect large bias, while hundreds of rolls reveal smaller flaws. The same arithmetic applies, because the amount of noise shrinks only with the square root of the number of observations.
Significance is a tool for being less fooled by luck, not a stamp of truth. Used with a planned sample, a clear question and a healthy skepticism, it keeps conclusions about your play within what the data can support.