Retrospective backtest
Scored after the fact against completed 2025 games. This is NOT the live forward ledger and must never be presented as one. For projections graded as they happen, with nothing known in advance, see the live forward record.
2025 season, scored after the fact
Shawn Childs' weekly projections for the complete 2025 season, graded against what actually happened. Because the games were already played when this was computed, it answers "how good were these numbers?" — not "how good are we live."
18 weekly TeamsSplits workbooks (weeks 1-18; week 17 was initially absent and supplied 2026-08-04) x the validated 2025 player-game ingest
Brier score measures probabilistic accuracy — lower is better, and 0 is perfect. Against a baseline that simply predicts each event's base rate (0.1747), these projections score 0.1367, a skill score of 0.218. Positive skill means the projections carried real information beyond knowing how often the event happens in general.
Expected vs observed
Every predicted probability, bucketed, against how often the event actually occurred. A perfectly calibrated forecast puts observed equal to predicted in every row.
| Predicted band | Avg predicted | Observed | Gap | n |
|---|---|---|---|---|
| 0–10% | 4.2% | 4.2% | +0.0 | 4,867 |
| 10–20% | 14.4% | 14.2% | -0.2 | 1,962 |
| 20–30% | 24.3% | 22.2% | -2.1 | 1,016 |
| 30–40% | 35.0% | 31.7% | -3.3 | 694 |
| 40–50% | 44.5% | 38.0% | -6.5 | 458 |
| 50–60% | 55.5% | 36.8% | -18.7 | 628 |
| 60–70% | 64.8% | 47.3% | -17.5 | 640 |
| 70–80% | 74.6% | 58.7% | -15.9 | 618 |
| 80–90% | 85.1% | 61.8% | -23.3 | 476 |
| 90–100% | 94.8% | 75.9% | -18.9 | 506 |
By position, with interval coverage
Coverage asks whether the uncertainty is honest: of the outcomes that should land inside a 50% interval, how many did? A well-calibrated interval covers close to its stated rate — higher is not better here, closer is.
| Position | Player-weeks | MAE (FP) | Brier | 50% interval | 80% interval |
|---|---|---|---|---|---|
| QB | 525 | 5.78 | 0.1709 | 49.7% | 81.1% |
| RB | 1,081 | 5.01 | 0.1308 | 49.0% | 78.2% |
| WR | 1,629 | 5.09 | 0.1270 | 51.6% | 81.1% |
| TE | 720 | 4.34 | 0.1426 | 48.6% | 78.9% |
By sample size
Absolute error rises with sample size because players with more prior weeks are higher-usage and carry larger projections. Relative error (MAE / mean projection) is the like-for-like comparison and moves the other way.
| Prior weeks on that player | n | Mean projection | MAE | Relative MAE |
|---|---|---|---|---|
| 0–2 prior weeks | 530 | 3.99 | 3.19 | 0.80 |
| 3–5 prior weeks | 1,097 | 9.52 | 4.79 | 0.50 |
| 6–9 prior weeks | 1,156 | 10.65 | 5.37 | 0.50 |
| 10–17 prior weeks | 1,172 | 11.72 | 5.73 | 0.49 |
Week by week
| Week | Player-weeks | MAE | Brier |
|---|---|---|---|
| Week 4 | 288 | 4.76 | 0.1315 |
| Week 5 | 251 | 4.96 | 0.1425 |
| Week 6 | 255 | 4.88 | 0.1326 |
| Week 7 | 269 | 5.19 | 0.1358 |
| Week 8 | 233 | 4.90 | 0.1433 |
| Week 9 | 244 | 5.18 | 0.1440 |
| Week 10 | 247 | 4.83 | 0.1309 |
| Week 11 | 266 | 5.40 | 0.1463 |
| Week 12 | 242 | 4.79 | 0.1305 |
| Week 13 | 276 | 4.92 | 0.1392 |
| Week 14 | 254 | 4.95 | 0.1406 |
| Week 15 | 275 | 5.33 | 0.1375 |
| Week 16 | 292 | 5.24 | 0.1432 |
| Week 17 | 285 | 4.92 | 0.1346 |
| Week 18 | 278 | 5.03 | 0.1191 |
Method
- · Probabilities are derived empirically, walk-forward: for week N and position P, the residual pool is (actual - projected) from weeks before N only.
- · P(actual >= threshold) = the share of that pool for which projected + residual clears the threshold. No distribution is assumed.
- · Weeks 1-3 are burn-in that builds the first pool and are not scored.
- · Predictive intervals use the same pool's quantiles; coverage is the share of outcomes falling inside.
- · Scoring: PPR for RB/WR/TE, Shawn's 4-point passing-TD scheme for QB.
- · Projected-but-did-not-play rows are excluded here and counted separately in the season report.
Weeks covered: 1–18 complete (weeks 4–18 scored after burn-in). Thresholds scored: QB 15/20/25 · RB 10/15/20 · WR 10/15/20 · TE 8/12/16 fantasy points. 11,865 threshold events in total.
This page is retrospective by construction and is kept separate from the live forward record on purpose. The two are never combined into a single claim.