Grading Our Own Board: How We Score Our Projections in Public

Three Measurements, Three Different Failures

A projection can fail in three separate ways, and no single number catches all three. The point estimate can land far from the actual total. The range around that estimate can be the wrong width. The whole board can be accurate and still tell you nothing the market was already telling you for free.

Grading means running all three tests and publishing all three answers, including the seasons and positions where the answers came back bad. The weekly rows, the graded pool, and the misses live on the accuracy ledger from Week 1; the backtest that is the board's only season-level record, and the inputs the projections are built from, sit on the methodology page. This piece is the scoring rule behind those numbers, and it carries no current results of its own, because current numbers move and belong to the surfaces that own them.

To make the arithmetic visible, take an invented stretch of running back board — six rows, numbers made up to show the shape, no real players and no real season. Projected medians of 210, 196, 174, 158, 141, and 122 points, in that order. The same six finished the year at 188, 241, 170, 96, 149, and 131.

Mean Absolute Error On The Median

Mean absolute error is the average distance between a projected median and the actual scored total, taken without regard to direction. A projection 30 points high and a projection 30 points low are the same size mistake, and letting them cancel would hand the board a way to hide.

Run it on the invented six. The gaps are 22, 45, 4, 62, 8, and 9 points, summing to 150 across six rows for a mean absolute error of 25.0. The median gap in the same set is 15.5, and most of that distance is one row, the fourth, missed by 62, pulling the average up behind it. The ledger prints the mean; the median gap is here to show how one blown call can own a good average.

The pool is stated before the results are. The weekly ledger grades every player we projected who took the field that week; a projected player who did not play is set aside rather than scored as a zero, and only players we actually projected that week are in the pool. At season level the backtest measures the board's order rather than its points, position by position; that is the third measurement.

Band Coverage And The Width Problem

Every projection ships with a range around its median, and the P10 to P90 band makes a testable claim: eight of every ten outcomes should land inside it. Coverage is the test. Count the graded pool, count how many final totals fell between their own P10 and P90, divide.

Bands on the invented six run 150 to 268, 138 to 252, 120 to 230, 104 to 208, 92 to 190, and 76 to 172. Five of the six finals landed inside. The fourth row finished at 96, under its P10 of 104, and that is the same row carrying the 62-point error one section up.

Five of six is 83 percent against a claimed 80, which reads like a pass and proves nothing. Coverage needs hundreds of rows before it says anything at all, and any coverage figure published without its sample size is decoration. The backtest's own figures come with theirs: across four held-out seasons of roughly 415 to 438 graded players each, the bands caught between 66.7 and 83.2 percent of outcomes against the 80 they claim, and in season the ledger prints the same share week by week in its range hit column.

Calibration fails in two directions, and only one of them feels like failure. Narrow bands drive coverage under 80 and mark the board as overconfident. Wide bands drive coverage to 95, make the board look careful, and quietly stop pricing anything; a floor-to-ceiling spread that covers every outcome is a way of declining to answer. The width test is what separates an uncertainty estimate from a hedge, which is why high-variance players get read on band width before anything else.

Rank Order Against Two Baselines

Point accuracy and usefulness are separate questions. A board that nails every total while ranking players in exactly market order has produced a precise copy of the room, and the room is available at no charge.

The third measurement is rank correlation: how closely the board's preseason order matched the order players actually finished in, scored as Spearman correlation, where 1 is a perfect match and 0 is no relationship at all. On its own that number is unreadable, so it is always scored beside two baselines answering the same question — last season's finishing order carried forward as this season's rank, and the market's order, ADP. A board earns its keep by beating both.

The invented six, ranked by projected median, run 1 through 6 in the order given. They finished 2, 1, 3, 6, 4, 5: the second row won the year and the fourth collapsed. Spearman works off the squared rank gaps — one, one, zero, four, one, one, summing to 8 — and lands at 0.77 for the board.

Say the market had them 1, 5, 4, 6, 2, 3; its squared gaps sum to 26, for 0.26. Say last year's finish had them 1, 3, 2, 5, 6, 4; that order scores 0.66. On this stretch the board beats both, and the row that cost it the most ground is the fourth: the largest point error, the only band break, and the one place the market sat closer, all on one player. That is the shape this example was built to show: the three tests failing together on one row and staying independent everywhere else.

The real record is thinner than the invented one. Three of the four held-out seasons went to the board over the repeat-last-year order, none by more than 0.0045 Spearman; the fourth, 2022, went the other way by 0.0063, on a model trained on a single prior season.

Against ADP the board won all three seasons with a market to grade, the closest by 0.0068. Split by position, quarterbacks, running backs, and tight ends beat the prior-season order on average and wide receivers did not, and the methodology page states the miss directly under the tables that carry the wins. Gap-versus-ADP as a drafting tool gets its own treatment on the ADP values and steals page; here it is only the third test.

Reading The Record

One season is one sample, and a board can post a good year on luck the same way a manager can. Three measurements across several seasons, split by position, is the smallest evidence that begins to separate process from outcome, and even that record describes the process while this year's results stay open.

The season model behind the draft board reruns by hand and ADP updates daily, so the board and the market both move underneath any number written in prose. The weekly ledger grades the first version posted before each week's kickoff and adds a row a week from Week 1; the preseason board itself has no live season grade, so its season-level record is the backtest and nothing else, and the ledger says that plainly instead of borrowing a number. That is why nothing here quotes a live rank, price, or gap.

Losing rows get the same precision as winning ones. A season where the bands ran narrow gets its coverage printed next to the seasons where they held, and the year the model lost to a plain prior-season order sits in the same table as the years it won.

This page is the scoring rule; the scores sit on the accuracy ledger, and the projections already on the record sit on the current rankings. Once the ledger has rows, start at its worst week rather than its best, and decide from there how hard to lean on any of it, this piece included.