Model Performance
Every FINAL game is scored against each model's pre-game prediction, built only from rating snapshots dated before tip-off. The Book Closing Line row is the sportsbook benchmark. Model predictions are for analytical purposes only and do not constitute betting advice.
Spread predicted home margin vs. actual
| Model | Games | MAE | RMSE | Winner % |
|---|---|---|---|---|
| Baseline (ML) | 5,531 | 9.37 | 11.91 | 70.6% |
| Baseline Plus (ML) | 5,425 | 9.29 | 11.79 | 70.6% |
| Model 3 (ML) | 5,531 | 9.41 | 12.00 | 70.9% |
| Model 4 (ML) | 5,293 | 9.18 | 11.66 | 70.6% |
| Massey | 5,531 | 9.53 | 12.10 | 70.4% |
| Adjusted Efficiency | 5,531 | 9.52 | 12.10 | 70.3% |
| Book Closing Line | 5,705 | 8.92 | 11.23 | 72.7% |
Total predicted combined score vs. actual
| Model | Games | MAE | RMSE | Bias |
|---|---|---|---|---|
| Baseline (ML) | 5,531 | 14.17 | 17.97 | +4.38 |
| Baseline Plus (ML) | 5,425 | 13.90 | 17.63 | +3.68 |
| Model 3 (ML) | 5,531 | 14.27 | 18.12 | +5.07 |
| Model 4 (ML) | 5,293 | 13.95 | 17.69 | +3.90 |
| Massey Totals | 5,531 | 14.10 | 17.74 | -1.28 |
| Adjusted Efficiency | 5,531 | 14.13 | 17.77 | -1.30 |
| Book Closing Line | 5,734 | 13.37 | 16.89 | +0.60 |
Win Probability predicted home-win chance vs. outcome
| Model | Games | Log Loss | Brier | Accuracy |
|---|---|---|---|---|
| Baseline (ML) | 5,531 | 0.5541 | 0.1881 | 70.8% |
| Baseline Plus (ML) | 5,425 | 0.5538 | 0.1880 | 70.6% |
| Model 3 (ML) | 5,531 | 0.5553 | 0.1885 | 70.6% |
| Model 4 (ML) | 5,293 | 0.5543 | 0.1883 | 70.9% |
| Massey | 5,531 | 0.5584 | 0.1887 | 70.4% |
| Bradley-Terry | 5,531 | 0.6246 | 0.2076 | 68.8% |
| Weighted Bradley-Terry | 5,531 | 0.6835 | 0.2157 | 68.7% |
| Adjusted Efficiency | 5,531 | 0.5581 | 0.1887 | 70.3% |
| Book Closing Line | 5,729 | 0.5129 | 0.1732 | 73.0% |
vs. Closing Line identical games only — the honest head-to-head
| Model | Games | Model MAE | Book MAE | Δ | ATS | O/U | CLV |
|---|---|---|---|---|---|---|---|
| Baseline (ML) | 5,487 | 9.37 | 8.89 | +0.47 | 50.6% (5,487) | 50.2% (5,513) | — |
| Baseline Plus (ML) | 5,381 | 9.29 | 8.86 | +0.43 | 50.5% (5,381) | 50.4% (5,407) | — |
| Model 3 (ML) | 5,487 | 9.41 | 8.89 | +0.51 | 50.3% (5,487) | 50.8% (5,513) | — |
| Model 4 (ML) | 5,249 | 9.18 | 8.87 | +0.31 | 50.2% (5,249) | 50.4% (5,275) | — |
| Massey | 5,487 | 9.53 | 8.89 | +0.64 | 50.3% (5,487) | — | — |
| Massey Totals | 0 | — | — | — | — | 49.6% (5,513) | — |
| Adjusted Efficiency | 5,487 | 9.52 | 8.89 | +0.63 | 50.2% (5,487) | 49.5% (5,513) | — |
All columns pair each model with the book closing line on the same games, so coverage differences can't skew the comparison. ATS/O-U count only games where the model disagrees with the closing number (pushes excluded) — 52.4% is breakeven at standard −110 vig. CLV counts games where the line moved off the opener and the model had taken a side against it; above 50% means the market tended to move toward the model. Not betting advice.
Month by Month do the models close the gap to the book as the season develops?
Monthly accuracy over the full season (the window selector does not apply here). Lower is better for both metrics; the dashed line is the book benchmark. Early-season months reflect ratings that are still converging on limited games.
By Conference where does the model beat the book?
| Conference | Games | Model MAE | Book MAE | Δ vs Book | Winner % |
|---|---|---|---|---|---|
| Big South Conference | 164 | 9.17 | 9.50 | -0.33 | 77.4% |
| Horizon League | 188 | 8.99 | 8.99 | -0.00 | 69.1% |
| Conference USA | 204 | 9.02 | 8.99 | +0.03 | 62.7% |
| Southeastern Conference | 354 | 10.08 | 9.94 | +0.14 | 75.4% |
| Southland Conference | 195 | 8.39 | 8.25 | +0.14 | 70.8% |
| Patriot League | 183 | 8.83 | 8.62 | +0.21 | 67.8% |
| Atlantic Sun Conference | 221 | 9.23 | 9.00 | +0.23 | 71.0% |
| Southern Conference | 179 | 8.75 | 8.51 | +0.23 | 70.4% |
| Big Sky Conference | 175 | 8.87 | 8.59 | +0.29 | 70.3% |
| Sun Belt Conference | 257 | 9.21 | 8.90 | +0.31 | 66.1% |
| Atlantic 10 Conference | 277 | 9.00 | 8.62 | +0.38 | 68.2% |
| Southwestern Athletic Conference | 222 | 9.68 | 9.29 | +0.39 | 67.1% |
| Mid-American Conference | 235 | 9.29 | 8.90 | +0.39 | 73.2% |
| Coastal Athletic Association | 240 | 9.01 | 8.62 | +0.40 | 70.0% |
| Big East Conference | 221 | 8.94 | 8.54 | +0.40 | 74.2% |
| Missouri Valley Conference | 207 | 9.87 | 9.47 | +0.40 | 70.0% |
| Big 12 Conference | 343 | 9.78 | 9.35 | +0.42 | 76.1% |
| Big West Conference | 193 | 9.37 | 8.90 | +0.47 | 67.9% |
| Mountain West Conference | 230 | 9.74 | 9.27 | +0.47 | 74.8% |
| Big Ten Conference | 368 | 9.98 | 9.48 | +0.50 | 76.9% |
| Summit League | 159 | 10.20 | 9.68 | +0.52 | 69.2% |
| Ivy League | 136 | 9.63 | 9.10 | +0.54 | 66.9% |
| West Coast Conference | 233 | 9.69 | 9.15 | +0.54 | 76.0% |
| Northeast Conference | 179 | 8.82 | 8.23 | +0.60 | 74.9% |
| American Conference | 251 | 8.72 | 8.11 | +0.60 | 72.5% |
| Atlantic Coast Conference | 373 | 9.05 | 8.44 | +0.62 | 74.8% |
| Western Athletic Conference | 125 | 8.80 | 8.16 | +0.64 | 72.0% |
| America East Conference | 167 | 9.91 | 9.24 | +0.66 | 74.3% |
| Metro Atlantic Athletic Conference | 231 | 9.34 | 8.65 | +0.69 | 69.7% |
| Ohio Valley Conference | 179 | 10.19 | 9.44 | +0.75 | 66.5% |
| Mid-Eastern Athletic Conference | 149 | 9.24 | 8.10 | +1.14 | 72.5% |
Paired comparison: only games where both the selected model and the book closing line made a spread prediction, sorted best-for-the-model first. Cross-conference games count toward both teams' conferences. Small samples swing wildly — read the Games column before drawing conclusions.
Calibration when a model says 70%, does the home team win 70% of the time?
Predictions are grouped into deciles of predicted home-win probability; each point compares the decile's average prediction (x) with the actual home-win rate (y). A perfectly calibrated model follows the dashed diagonal.