ML ceiling check, row level
A fair comparison between the simple model and two machine-learning models. Random Forest and XGBoost got exactly the same task as the simple model (one row per stay and candidate), the raw inputs plus the place type, and exactly the same grouped test sets.
| Model | In-sample | CV (grouped) | CV building | CV street | CV open space |
|---|---|---|---|---|---|
| Logit, Final model | 0.721 | 0.716 ± 0.076 | 0.896 | 0.429 | 0.491 |
| Random Forest | 0.762 | 0.699 ± 0.063 | 0.942 | 0.439 | 0.277 |
| XGBoost | 0.849 | 0.695 ± 0.071 | 0.893 | 0.461 | 0.371 |
Both tree models win on the training data (in-sample), and both lose on unseen data (CV). Almost the whole difference comes from open space. For building and street, the three models are close. So with a fair test, the machine-learning models offer no improvement. This reverses the earlier result in ML baseline, stay level (superseded).
Settings: Random Forest with 400 trees, depth 8, at least 5 rows per leaf. XGBoost with 300 rounds, depth 4, learning rate 0.1. Neither was tuned. All three models find the same test sets hard (Fold difficulty).
The clearest case of Overfitting signature. Which inputs the trees use: Sigma in tree importance. Part of Complexity and transfer.