QMML Christmas Hackathon — return prediction, and a leak I did not catch

Customer-, product- and category-level return rates were computed on the full training set, so every training row's feature contained its own label. The test features were built correctly from training statistics — which is exactly why the training fit and the test behaviour were different models.

What was submitted

Three groups of features, seventeen in total, chosen over a baseline with 55+ one-hot columns:

The model was a scaled logistic regression, trained once on all 8,000 rows, and the submission was its predictions on the 2,000 test rows.

The leak

The aggregation function takes a reference frame for the group statistics. For the test set it was called with the training frame, which is correct. For the training set it was called with no reference, so the training frame was used to encode itself:

# As written in the competition notebook
def add_aggregation_features(df, train_df=None):
    reference = train_df if train_df is not None else df
    customer_rates = reference.groupby('CustomerID')['ReturnFlag'].mean().to_dict()
    df['CustomerReturnRate'] = df['CustomerID'].map(customer_rates).fillna(0.5)
    ...

train_enhanced = add_aggregation_features(train_df)            # reference = train_df itself
test_enhanced  = add_aggregation_features(test_df, train_df)   # correct direction

For a customer with k transactions in the training set, CustomerReturnRate on each of those rows is the mean of k labels, one of which is the row's own. With k = 1 the feature is the label. With small k it is close. The model therefore learned a relationship between the aggregate and the target that is far stronger than anything the test rows can offer, where the aggregate is built from other people's transactions only. The same applies to the product- and category-level rates, more weakly as the groups get larger.

Two consequences. First, any internal validation on those features would have been optimistic — but there was none: cross_val_score is imported in the notebook and never called, so the leaderboard was the only evaluation. Second, the submission's test predictions were produced by a model whose coefficients were fitted to leaked features, so the 2nd-place score was earned despite the construction, not because of it. How much better a clean version would have scored is unknown.

The correct construction

Target statistics have to be computed out of fold: every training row's encoding must come from rows it is not part of, and the test encoding from the full training set.

# The construction the notebook should have used (not the competition implementation)
from sklearn.model_selection import StratifiedKFold

def oof_target_encode(train, test, col, target, n_splits=5, smoothing=10, seed=0):
    prior = train[target].mean()
    def encode(fit_df, apply_df):
        stats = fit_df.groupby(col)[target].agg(['sum', 'count'])
        enc = (stats['sum'] + prior * smoothing) / (stats['count'] + smoothing)
        return apply_df[col].map(enc).fillna(prior)
    train_enc = pd.Series(index=train.index, dtype=float)
    for fit_idx, enc_idx in StratifiedKFold(n_splits, shuffle=True, random_state=seed).split(train, train[target]):
        train_enc.iloc[enc_idx] = encode(train.iloc[fit_idx], train.iloc[enc_idx]).values
    return train_enc, encode(train, test)

A synthetic check makes the size of the problem concrete. On 8,000 rows with roughly 3,000 customers and labels drawn at random — so there is nothing to learn — the notebook's construction produces a customer feature with correlation 0.59 to the label. The out-of-fold construction above produces 0.00. A model trained on the first would report strong validation performance on pure noise.

The smoothing term matters on this data: many customers have a handful of transactions, and an unsmoothed mean of two labels is mostly noise. Model selection then needs its own outer cross-validation, with the encoding refitted inside every fold, or the same leak returns through the back door.

Two months later, in the Valentine's hackathon, the target encoding was built exactly this way.

What still stands

• The placement. The organisers scored the test predictions; the flaw hurt rather than helped.
• The temporal features are leak-free and encode a real hypothesis about gift returns after 25 December.
• A linear model with a small, deliberate feature set was competitive against wider one-hot baselines, which is a useful prior for small tabular problems.

Limitations

What I would do next

• Re-run with out-of-fold encoding inside 5-fold CV and report the contribution of each feature group by ablation.
• Compare the logistic model with a gradient-boosted tree under the same protocol — the "simple model wins" claim deserves a controlled test.
• Save the validation score and the leaderboard score in the repository, so the next reader does not have to take the placement on trust.

View the repositoryAll work