Case study · Tabular ML · Solo entry
QMML Valentine's Hackathon — weak-signal classification at scale
Predict whether a survey respondent has a Valentine's date from demographic and social attributes: 700,000 training rows, ROC-AUC, and a leaderboard where the top entries were separated by less than half a thousandth. An AUC of 0.59 is not a poor model here; it is what the signal allows, and the competition was about extracting it cleanly.
- Result
- 2nd place · 0.59235 AUC (1st: 0.59275, a gap of 0.0004), QMUL Machine Learning Society, February 2026
- Data
- ~700k training rows, Kaggle-hosted; demographic, lifestyle and survey-timing columns; several high-cardinality categoricals
- Method
- Out-of-fold target encoding → LightGBM, XGBoost and CatBoost each tuned with 50 Optuna trials → 5-fold out-of-fold predictions → rank-averaged ensemble
- Stack
- Python, pandas, scikit-learn, LightGBM, XGBoost, CatBoost, Optuna, SciPy; GPU training on Colab
- Code
- github.com/M-Rodani1/qmml-valentines-hackathon
0.59235 against a winner at 0.59275. The narrow range says the ceiling was low for everyone; the ranking came from encoding and validation discipline, not from a bigger model.
Reading the number
An AUC of 0.5 is a coin flip; 0.59 means the model ranks a random positive above a random negative 59% of the time. On a dataset of survey attributes predicting a personal outcome, that is close to the information the features contain. The practical evidence for a low ceiling is the leaderboard: with hundreds of thousands of rows, differences of 0.0004 between strong entries are not noise, but they are the last fraction of a signal everyone had largely found. The job was to lose none of it to leakage or to badly calibrated combination.
Pipeline
- Features
Survey_Date→ month, hour, day of week; BMI, log income, income per age; four interaction products; six missingness flags and a missing-count - Target encoding8 categoricals, 5-fold out of fold, global-mean fallback
- TuningOptuna, 50 trials per model, on one held-out fold with early stopping
- OOF predictions5-fold stratified CV per model; test predictions averaged over folds
- Ensemblerank-transform each model, average, submit
Target encoding, out of fold
Eight categorical columns — zodiac sign, pets, favourite colour, social-media presence, gender, education, job type, location type — were replaced by the mean target within each level, computed with a 5-fold stratified split so that each training row's value came from the other four folds. Unseen levels fell back to the global mean, and the test set was encoded with statistics from the full training set. The label-encoded originals were kept alongside, so the tree models could use either representation.
skf_te = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
for col in target_encode_cols:
train[f'{col}_te'] = np.nan
for tr_idx, val_idx in skf_te.split(train, train['Has_Valentine']):
col_means = train.iloc[tr_idx].groupby(col)['Has_Valentine'].mean()
train.loc[train.index[val_idx], f'{col}_te'] = (
train.iloc[val_idx][col].map(col_means).fillna(global_mean))
test[f'{col}_te'] = test[col].map(train.groupby(col)['Has_Valentine'].mean()).fillna(global_mean)This is the construction the Christmas hackathon two months earlier got wrong. On a weak-signal problem the difference is not academic: an in-sample encoding would have produced a model that looked excellent under any internal check and collapsed on the leaderboard.
Models, tuning and ensemble
| Model | Tuned parameters | Fixed |
|---|---|---|
| LightGBM | num_leaves 31–256, learning rate 0.01–0.1 (log), min_child_samples 20–200, feature and bagging fractions 0.5–1.0, bagging_freq 1–7, L1 and L2 (log) | 1,000 rounds, seed 42 |
| XGBoost | max_depth 3–10, learning rate 0.01–0.1 (log), subsample and colsample 0.5–1.0, min_child_weight 1–20, alpha, lambda, gamma | hist tree method, 1,000 rounds |
| CatBoost | depth 4–10, learning rate 0.01–0.1 (log), l2_leaf_reg 1–10 (log), bagging temperature, random strength | 1,000 iterations, AUC eval metric |
Each tuned model was then refitted five times under a stratified 5-fold split to produce out-of-fold predictions for the whole training set and fold-averaged predictions for the test set. The three prediction vectors were converted to ranks (each divided by its length) and averaged. Rank averaging was chosen because the three libraries produce differently calibrated probabilities on the same data; averaging raw probabilities lets the model with the widest spread dominate, while averaging ranks weights each model's ordering equally — and AUC is a statement about ordering.
Limitations
The notebook was run on Colab and its outputs were not saved, so the per-model and ensemble out-of-fold AUCs are not in the repository. The 0.59235 is the organisers' leaderboard score.
Hyperparameters were tuned on the first fold of the same 5-fold split later used for out-of-fold scoring, so the OOF estimate for that fold is mildly optimistic. A nested split, or a separate tuning holdout, would remove that.
The target-encoding folds (random_state=0) and the model folds (random_state=42) do not coincide. The test predictions are unaffected, but a training row's encoding can include rows that later sit in the same model validation fold, which slightly inflates the internal CV estimate.
The rank ensemble uses equal weights. With saved OOF predictions, weights could have been fitted — cheaply, and with a small expected gain on a problem this close.
What I would do next
- Persist the OOF predictions and scores; fit ensemble weights on them and report the gain, or lack of it, over equal weights.
- Align the encoding and model folds, and move tuning to a separate holdout.
- Add a plain logistic-regression baseline on the encoded features to show how much the boosters actually add on a weak-signal problem.