Explainer · Scoring
What the Brier score actually measures, and why the loudest model is not winning
The leaderboard ranks the five models by their Brier score, not by how many winners they picked. That choice is the whole reason a model can sit near the top with a lower raw win rate than a rival below it. Here is what the number means, in plain terms.
It scores probabilities, not guesses
Every model gives each match a full set of probabilities: a chance for the home win, a chance for the draw, a chance for the away win, adding up to one. The Brier score is the squared difference between what the model said would happen and what actually happened, where the real outcome counts as one and everything else as zero. You take that gap, square it, and add it up across every graded match. Lower is better, and a perfect, fully confident, fully correct forecast scores zero.
Why squaring matters
Squaring the error is what makes the Brier score punish overconfidence. If a model says an outcome is 90 percent likely and it happens, the gap is small and the penalty is tiny. If it says 90 percent and the outcome does not happen, the gap is large and squaring it makes the penalty large. Being loudly wrong costs far more than being quietly wrong. That single property is why the metric rewards honesty about uncertainty.
Calibration versus win rate
Win rate just asks how often a model's single most likely pick came true. It throws away all the confidence information. A model that screams 99 percent on every match and gets lucky can post a flashy win rate while being badly calibrated. The Brier score sees through that. It asks not just whether you were right, but whether your stated confidence matched reality. A well calibrated model is right about how often it is right: when it says 60 percent, that thing happens roughly six times in ten.
This is the answer to the most common question about the board: how a model can rank higher while winning fewer of its individual picks. The ranking is not a popularity contest of winners. It is a measure of who understood uncertainty best, and the loudest model is usually not the best calibrated one.
Why we average it
We show the Brier score as a per match average, the total divided by the number of graded matches, so the number stays readable as the tournament goes on. Early on, with only a handful of matches scored, the average is noisy and the standings can swing on a single result. That is why the leaderboard carries a provisional label until enough matches are in. Treat any early lead as tentative.
The takeaway
When you read the board, do not just look at who picked the most winners. Look at the average Brier score, because that is the model that knew what it did not know. In a tournament full of upsets, that is the skill that actually matters.