Three questions analytics tries to answer
Almost every number published about a competitive game answers one of three questions. How good is this player or team? How sure are we about that? And what is likely to happen in the next match?
The first question is the job of ratings. The second is about uncertainty and sample size. The third is the job of win probabilities, which can be calculated before a match starts or updated while it is being played. The rest of this article follows that chain, and shows where each link can break.
Ratings: turning results into a number
A rating system takes a history of wins and losses and compresses it into a single value per player. The key idea is that the number is only meaningful relative to other ratings. A gap between two ratings maps to an expected result, and that expectation is checked against what actually happens.
The classic example is the Elo rating, originally designed for chess. Each player has a number, the difference between two numbers gives an expected score, and after a match the ratings shift in proportion to how surprising the result was. Beating a much higher-rated opponent moves your rating a lot, while beating a much lower-rated one moves it very little.
Elo has one weakness: it treats every rating as equally trustworthy. A player with three games behind them and a veteran with three thousand are described by the same kind of number. Systems such as Glicko and the TrueSkill ratings family add an uncertainty estimate to each player, and TrueSkill also handles team games where one result has to be credited across several people.
Stats: describing how a player performs
Ratings only use results. Stats describe what happened inside the results, such as kills, damage dealt, objective time, gold earned or accuracy. They are useful because they carry more information per match than a single win or loss.
They also come with traps:
- Totals depend on how long a match lasted, so rates (per round, per minute) are usually fairer than raw counts.
- Roles differ. A support and a carry should not be compared on the same stat line.
- Stats are shaped by teammates and opponents, so a strong number may reflect a good situation rather than a good player.
- The more specific a stat is, the fewer events it rests on.
That last point is why player statistics need to be read with their sample size in mind. A figure built on a handful of matches can look dramatic and still be mostly noise.
From ratings to win probabilities
Once ratings exist, they can be turned into a forecast. With Elo-style ratings, a 100 point advantage corresponds to an expected score of roughly 0.64, and 200 points to roughly 0.76. These are probabilities of winning a single game, and the usual simplification is that the games are independent and the gap stays fixed.
Real contests are rarely a single game. Series length changes how often the stronger side wins, because a longer series gives skill more chances to show up. Tournament formats such as single elimination, round robin and Swiss matter for the same reason: each format filters out luck to a different degree.
During a match, the forecast can be updated. A win probability model combines the pre-match estimate with the current state, such as the score, the clock and resources held, and outputs a new probability after each event.
Telling skill from noise
Results contain both skill and chance. Several ideas help separate them.
- Extreme performances are partly luck, so they tend to be followed by more ordinary ones. This is regression to the mean, and it explains why a hot streak or a dreadful week is a weak predictor of the next one.
- The law of large numbers says averages settle down as the number of observations grows, which is why long records are more informative than short ones.
- Confidence intervals put a range around an observed win rate instead of treating it as exact.
- Statistical significance asks whether a gap between two groups could plausibly be produced by chance alone.
Together these explain much of the old question of luck and skill: skill shows through reliably only when there are enough games for chance to average out.
Checking whether a model is any good
A forecast can be tested. If a model says 70% for a hundred different matches, roughly seventy of those favourites should win. Systematic departures from that, such as favourites winning only 60% of the time, suggest the model is overconfident.
A common scoring rule is the Brier score, the squared gap between the forecast and the result coded as 1 for a win and 0 for a loss. A 0.7 forecast on a winner scores (0.7 - 1)^2 = 0.09, while a 0.9 forecast on a loser scores 0.9^2 = 0.81. Lower is better, and the penalty for confident misses is severe. Averaging this over many matches shows whether one model beats another, though a small number of matches again gives a noisy answer.
Limits worth remembering
Ratings and probabilities describe the past and assume the future resembles it. Game patches change what is strong, rosters change, and players improve or lose form. A rating also only reflects the opponents someone actually faced, so players in separate pools are hard to compare.
A probability is not a promise. A 75% favourite loses about one match in four, and nothing in these methods guarantees a particular result or provides a reliable way to profit from predictions. They are best used as a way to think clearly about evidence, not to remove uncertainty.