Kaggle Playground S5E11 — loan-default prediction · case study

Baseline 0.923351 ± 0.000664 CV. Two ideas made it worse; one made it 0.00033 better. The gap to the top of the leaderboard was about 0.01.

Protocol

The validation was decided before the first model: five stratified folds, fixed seed, out-of-fold AUC reported as mean ± standard deviation across folds. Every experiment below used the same folds, so differences of a few ten-thousandths are comparable — and, at 594k rows, meaningful. The leaderboard was checked rarely and never used to choose between ideas.

Experiments

Every experiment, in order
#	Hypothesis	Experiment	CV AUC	Δ vs baseline	Decision
1	A tuned single booster is a strong baseline on 11 clean features	LightGBM, Optuna 30 trials over depth, leaves, learning rate, sampling	0.923351 ± 0.000664	—	Keep
2	Domain ratios (debt burden, payment capacity) add information	10 engineered features: net income, payment-to-income, risk score, …	0.922625	−0.000726	Drop
3	Combining LightGBM, XGBoost and CatBoost reduces variance	Six configurations, two blends (similar and deliberately extreme hyperparameters)	0.922904 / 0.922999	−0.000447 / −0.000352	Drop
4	Seed averaging reduces variance; statistics from the original data add information	Five LightGBM seeds averaged + 18 per-category target statistics from the 20k original rows	0.923682	+0.000331	Submit

The final submission was experiment 4; its private leaderboard score was 0.92450, above the CV estimate by about 0.0008, which is consistent with the synthetic test set being slightly easier than the folds rather than with any leakage.

Why the engineered features hurt

The ratios were near-duplicates of columns the model already had. net_income correlated with annual_income at r = 0.987; the debt-interest interaction correlated with debt_to_income at r = 0.950. A gradient-boosted tree learns a monotone transform of an existing column for free through its splits, so a feature that is a smooth function of one or two inputs adds capacity without adding information — and here it added enough noise to cost 0.0007 AUC. The cheaper order of operations would have been to check correlations before running two hours of cross-validation.

Why the ensemble hurt

The out-of-fold predictions of the three libraries were correlated at ρ ≈ 0.991 and 0.994 for the two blends. The usual explanation is the variance of an average of M equally correlated predictors, ρσ² + (1 − ρ)σ²/M: with ρ that high, averaging removes well under one percent of the variance, and the small calibration differences between libraries are enough to tip the blend below the best single model. That is the theory; the measured CV drop is the evidence. In this dataset, with 11 features and a clean synthetic target, three boosting libraries converged on nearly the same ranking, and changing hyperparameters did not make them disagree. I would not generalise that beyond datasets like this one.

What helped

Two changes, combined in one run. Averaging five LightGBM seeds (individual CVs 0.923035–0.923323) trades five times the training cost for a small variance reduction. The external features are per-category mean, standard deviation and count of the target in the original 20,000-row dataset the competition data was synthesised from — a target encoding whose statistics come from a different sample, so they carry no leakage from the competition labels and, by the notebook's own check, almost no correlation with the existing numeric features (maximum |r| ≈ 0.01). Together: 0.923682, +0.00033 over the baseline. Small, but real at this sample size, and in the right direction for the right reasons.

# Multi-seed averaging around the same 5-fold protocol
for seed in [42, 43, 44, 45, 46]:
    oof, test_pred, cv = train_lgbm_single_seed(X, y, X_test, LGBM_PARAMS, seed, n_splits=5)
    all_oof.append(oof); all_test.append(test_pred)
oof_final  = np.mean(all_oof, axis=0)
test_final = np.mean(all_test, axis=0)
cv_final   = roc_auc_score(y, oof_final)   # 0.923682

Limitations

Scope of the conclusions

All four conclusions are about this dataset. "Correlated predictions do not ensemble" and "ratio features do not help trees" are observations here, not rules; both are routinely false elsewhere.

The information-theoretic framing in the repository README (external statistics carrying positive conditional mutual information, engineered transforms carrying none) is an explanation, not a measurement — no mutual information was computed. The measured quantities are the correlations and the CV deltas.

The Optuna search was 30 trials; a longer search or a different objective might change the baseline slightly and, with it, the deltas.

Top 25% is a respectable but ordinary result; the top of the leaderboard was around 0.935.

What I would do next

Run the correlation and duplication checks as a fixed pre-flight step, before any model, and keep them in the repository.
Test ensemble diversity properly: a linear model and a neural network on the same folds, which would actually disagree with the boosters.
Report the leaderboard score of each submitted phase, not just the final one, so CV-to-leaderboard gaps can be tracked per change.

View the repository All work