Model Performance
Every FINAL game is scored against each model's pre-game prediction, built only from rating snapshots dated before tip-off. The Book Closing Line row is the sportsbook benchmark. Model predictions are for analytical purposes only and do not constitute betting advice.
Spread predicted home margin vs. actual
| Model | Games | MAE | RMSE | Winner % |
|---|---|---|---|---|
| Baseline (ML) | 5,531 | 9.37 | 11.91 | 70.6% |
| Baseline Plus (ML) | 5,425 | 9.29 | 11.79 | 70.6% |
| Model 3 (ML) | 5,531 | 9.41 | 12.00 | 70.9% |
| Massey | 5,531 | 9.53 | 12.10 | 70.4% |
| Book Closing Line | 5,705 | 8.92 | 11.23 | 72.7% |
Total predicted combined score vs. actual
| Model | Games | MAE | RMSE | Bias |
|---|---|---|---|---|
| Baseline Plus (ML) | 5,425 | 13.90 | 17.63 | +3.68 |
| Baseline (ML) | 5,531 | 14.17 | 17.97 | +4.38 |
| Model 3 (ML) | 5,531 | 14.27 | 18.12 | +5.07 |
| Massey Totals | 5,531 | 14.10 | 17.74 | -1.28 |
| Book Closing Line | 5,734 | 13.37 | 16.89 | +0.60 |
Win Probability predicted home-win chance vs. outcome
| Model | Games | Brier | Accuracy |
|---|---|---|---|
| Baseline (ML) | 5,531 | 0.1881 | 70.8% |
| Baseline Plus (ML) | 5,425 | 0.1880 | 70.6% |
| Model 3 (ML) | 5,531 | 0.1885 | 70.6% |
| Bradley-Terry | 5,531 | 0.2076 | 68.8% |
| Weighted Bradley-Terry | 5,531 | 0.2157 | 68.7% |
| Book Closing Line | 5,503 | 0.1779 | 72.3% |
vs. Closing Line identical games only — the honest head-to-head
| Model | Games | Model MAE | Book MAE | Δ | ATS | O/U | CLV |
|---|---|---|---|---|---|---|---|
| Baseline (ML) | 5,487 | 9.37 | 8.89 | +0.47 | 50.6% (5,487) | 50.2% (5,513) | — |
| Baseline Plus (ML) | 5,381 | 9.29 | 8.86 | +0.43 | 50.5% (5,381) | 50.4% (5,407) | — |
| Model 3 (ML) | 5,487 | 9.41 | 8.89 | +0.51 | 50.3% (5,487) | 50.8% (5,513) | — |
| Massey | 5,487 | 9.53 | 8.89 | +0.64 | 50.3% (5,487) | — | — |
| Massey Totals | 0 | — | — | — | — | 49.6% (5,513) | — |
All columns pair each model with the book closing line on the same games, so coverage differences can't skew the comparison. ATS/O-U count only games where the model disagrees with the closing number (pushes excluded) — 52.4% is breakeven at standard −110 vig. CLV counts games where the line moved off the opener and the model had taken a side against it; above 50% means the market tended to move toward the model. Not betting advice.
Month by Month do the models close the gap to the book as the season develops?
Monthly accuracy over the full season (the window selector does not apply here). Lower is better for both metrics; the dashed line is the book benchmark. Early-season months reflect ratings that are still converging on limited games.
By Conference where does the model beat the book?
| Conference | Games | Model MAE | Book MAE | Δ vs Book | Winner % |
|---|---|---|---|---|---|
| Big South Conference | 172 | 9.25 | 9.33 | -0.07 | 78.5% |
| Horizon League | 192 | 9.14 | 9.06 | +0.08 | 71.4% |
| Atlantic Sun Conference | 229 | 9.06 | 8.85 | +0.21 | 73.4% |
| Southeastern Conference | 364 | 10.13 | 9.86 | +0.27 | 75.8% |
| Southwestern Athletic Conference | 229 | 9.57 | 9.27 | +0.30 | 70.7% |
| Conference USA | 216 | 9.23 | 8.93 | +0.30 | 63.0% |
| Sun Belt Conference | 268 | 9.12 | 8.80 | +0.32 | 67.9% |
| Big Sky Conference | 184 | 8.85 | 8.40 | +0.45 | 70.1% |
| Southland Conference | 205 | 8.71 | 8.24 | +0.47 | 68.3% |
| Mountain West Conference | 238 | 9.67 | 9.17 | +0.50 | 74.8% |
| Big 12 Conference | 353 | 9.78 | 9.26 | +0.53 | 76.5% |
| Southern Conference | 184 | 9.05 | 8.52 | +0.54 | 71.7% |
| Big West Conference | 200 | 9.36 | 8.81 | +0.54 | 68.0% |
| Missouri Valley Conference | 213 | 10.05 | 9.49 | +0.56 | 69.0% |
| Northeast Conference | 184 | 8.88 | 8.29 | +0.59 | 73.9% |
| Atlantic 10 Conference | 290 | 9.30 | 8.70 | +0.60 | 68.3% |
| American Conference | 262 | 8.92 | 8.31 | +0.61 | 71.4% |
| Big East Conference | 228 | 9.31 | 8.69 | +0.62 | 76.3% |
| Mid-American Conference | 243 | 9.56 | 8.92 | +0.64 | 71.6% |
| America East Conference | 174 | 9.93 | 9.24 | +0.69 | 74.1% |
| Summit League | 166 | 10.34 | 9.64 | +0.70 | 68.7% |
| Patriot League | 193 | 9.34 | 8.62 | +0.72 | 67.9% |
| West Coast Conference | 243 | 9.79 | 9.05 | +0.74 | 74.9% |
| Western Athletic Conference | 132 | 8.89 | 8.11 | +0.79 | 71.2% |
| Atlantic Coast Conference | 384 | 9.35 | 8.55 | +0.80 | 74.0% |
| Coastal Athletic Association | 251 | 9.50 | 8.69 | +0.81 | 66.9% |
| Big Ten Conference | 383 | 10.21 | 9.39 | +0.82 | 77.3% |
| Ohio Valley Conference | 185 | 10.33 | 9.49 | +0.84 | 64.3% |
| Mid-Eastern Athletic Conference | 153 | 9.04 | 8.19 | +0.85 | 70.6% |
| Metro Atlantic Athletic Conference | 238 | 9.56 | 8.68 | +0.88 | 71.0% |
| Ivy League | 146 | 10.34 | 9.23 | +1.11 | 69.2% |
Paired comparison: only games where both the selected model and the book closing line made a spread prediction, sorted best-for-the-model first. Cross-conference games count toward both teams' conferences. Small samples swing wildly — read the Games column before drawing conclusions.
Calibration when a model says 70%, does the home team win 70% of the time?
Predictions are grouped into deciles of predicted home-win probability; each point compares the decile's average prediction (x) with the actual home-win rate (y). A perfectly calibrated model follows the dashed diagonal.