💧 Data Leakage
Data Leakage
[[cross-validation]] gives you an honest estimate — provided the folds are honest. Leakage is what makes them dishonest.
A leak is any path by which information from outside a model's legitimate training data reaches it. The symptom is always the same and always seductive: validation scores that are too good, and a model that disappoints in production. Nothing errors. Nothing warns you. The number simply is not measuring what you think it is.
This page works through five kinds of leak. Each one is built deliberately, measured against the honest baseline, and then fixed — so the size of the optimism is a number rather than a warning.
Display helpers ready: banner, step, check, expand, show_matrix, show_matmul, show_elementwise. ============================================================================== A DATASET WITH NO SIGNAL IN IT ============================================================================== 100 samples, 500 features, all drawn from N(0,1) labels: [50, 50] -- exactly balanced, shuffled X and y are INDEPENDENT. There is nothing to learn. Honest accuracy for any model whatsoever = 0.50 So anything above 0.50 on this page is a leak, measured. The honest baseline ============================================================================== proper pipeline, 5-fold : [0.55 0.5 0.45 0.45 0.45] mean = 0.4800 (truth 0.5000)
1. Preprocessing before splitting
The most common leak, and the most innocent-looking. You standardise the whole feature matrix in one step, and then hand the result to cross-validation.
The scaler computed its mean and standard deviation from every row, including the ones that will be test rows in each fold. Each fold's test data helped determine how its own training data was transformed.
With well-behaved features and plenty of rows this is a small effect. With many features relative to samples it is not small at all. The same applies to any fitted transformer — imputers, PCA, feature selection, target encoding — and feature selection is by far the worst offender, because choosing "the 5 features most correlated with y" using all the data is choosing the features that best fit the test rows.
==============================================================================
LEAK 1: FITTING A TRANSFORMER BEFORE THE SPLIT
==============================================================================
Scaling: fit on everything, then cross-validate
==============================================================================
scaled-then-split : 0.4800
pipeline (honest) : 0.4800
difference : +0.0000
Worth saying plainly: on this data that difference is essentially
nothing. Standardisation leaks two numbers per feature -- a mean and
a standard deviation, each estimated from all 100 rows -- and a test
row's influence on its own scaling is about 1/100 of it.
So this is a real but tiny leak. Fix it because it is free to fix,
not because it is what goes wrong. The next one is what goes wrong.
Feature SELECTION before the split: the same mistake, catastrophic
==============================================================================
select-then-split (leaky) : [0.65 0.45 0.7 0.6 0.85] mean 0.6500
selection inside pipeline : [0.6 0.35 0.5 0.6 0.3 ] mean 0.4700
truth : 0.5000
optimism from leaky selection: +0.1500
Picking the 5 features most associated with y, using all 100 rows,
then testing on rows that helped pick them. Out of 500 random
features some will correlate with y by chance -- and the selector
finds exactly those. This is how a model 'discovers' signal in noise.
Why the pipeline fixes it
==============================================================================
A Pipeline refits every step inside each fold, on that fold's
training rows only. The test rows of each fold are transformed by
statistics they had no part in computing.
approach CV honest?
------------------------------------------------------
fit_transform(X) then cross_val 0.6500 no
Pipeline(steps) in cross_val 0.4700 yes
------------------------------------------------------
2. Target leakage
A feature that could not exist at prediction time, or that was computed using the answer. This one is not a coding mistake — the code is correct, the pipeline is clean, the cross-validation is proper. The data is wrong.
Typical shapes:
discharge_diagnosisused to predict admission diagnosistotal_spendused to predict churn, where churned customers stop spending- a timestamp column that is only populated for positive cases
- an ID that was assigned in label order
The tell is a feature that is too predictive. A single column carrying 0.95 AUC in a problem where domain experts manage 0.70 is not a breakthrough.
==============================================================================
LEAK 2: A FEATURE THAT ENCODES THE ANSWER
==============================================================================
A realistic feature table: 10 candidate measurements, none informative
==============================================================================
10 features, 100 rows, no signal -> CV 0.5400
Now add one 'innocent' column derived from the target
==============================================================================
new column = y + N(0, 0.45)
correlation with target: 0.7544
In a real table this would be something like 'days_to_discharge' or
'total_spend' -- a genuine measurement, recorded AFTER the outcome.
CV with the leaked column : 0.8500
CV without it : 0.5400
truth : 0.5000
The cross-validation is textbook-correct. The pipeline is correct.
Every fold is honest. And the answer is still wrong, because the
information was already inside the feature matrix.
How you would catch it: score every feature on its own
==============================================================================
feature single-feature CV
----------------------------------
feature 0 0.6300
feature 1 0.5300
feature 2 0.4600
feature 3 0.4700
feature 4 0.5300
feature 5 0.4900
feature 6 0.4400
feature 7 0.4600
feature 8 0.5100
feature 9 0.5700
THE NEW ONE 0.9100
----------------------------------
the 10 genuine features: mean 0.5090, max 0.6300
the leaked column : 0.9100
it beats the best honest feature by +0.2800
A single column that outscores every other one, in a problem where
nothing should predict anything, is the signature of target leakage.
3. Group leakage
Covered from the splitting side in [[cross-validation]]; here is what it costs.
When several rows describe the same underlying entity — patient, session, device, customer — those rows are not independent. A random split puts some of an entity's rows in training and others in test, and the model can score by recognising the entity instead of learning the phenomenon.
The dangerous property is that this leak survives a perfectly correct pipeline. Nothing is fitted on the wrong data. The split itself is the flaw.
============================================================================== LEAK 3: THE SAME ENTITY ON BOTH SIDES OF THE SPLIT ============================================================================== Build data where the entity is recognisable and the label is per-patient ============================================================================== 30 patients x 4 recordings = 120 rows, 20 features each patient has a distinctive signature in the features the label is assigned per PATIENT, at random -- all 4 recordings from one patient share it, exactly as a clinical outcome would the label is independent of the features, so honest accuracy = 0.50 Random split vs grouped split ============================================================================== random 5-fold : [1. 1. 0.875 0.9583 0.875 ] mean 0.9417 group 5-fold : [0.4583 0.5417 0.5833 0.125 0.25 ] mean 0.3917 truth : 0.5000 optimism from ignoring patients: +0.5500 A 3-NN model finds the other recordings from the same patient sitting in the training set, and copies their labels. That is not generalisation, it is a lookup -- and at deployment every patient is new, so the lookup table is empty.
4. Temporal leakage
If your data is ordered in time, a random split trains on the future. Even without an explicit date column, leakage arrives through features computed over windows that straddle the split, or through normalising by a global statistic that only becomes knowable later.
The honest protocol is the one you will actually run in production: train on a prefix, predict the block after it, never the reverse.
5. Duplicate rows
The simplest leak of all. If the same row appears twice and the split separates the copies, the model has literally seen the test answer. Deduplicate before splitting — and note that near-duplicates (the same event logged twice, an augmented copy, a resampled row) do the same damage while being invisible to an exact-match check.
The demonstration below is deliberately blunt: duplicating every row takes a no-signal dataset to near-perfect accuracy.
============================================================================== LEAK 4 & 5: TIME, AND DUPLICATES ============================================================================== Temporal: random split vs forward-chaining ============================================================================== shuffled 5-fold : R2 mean +0.2522 forward-chaining : R2 mean -0.5813 gap : +0.8335 Shuffling lets the model interpolate between surrounding time points. Forward-chaining forces it to extrapolate, which is the actual task. The second number is the one that predicts production behaviour. Duplicates: the bluntest leak there is ============================================================================== with every row duplicated : 0.9200 deduplicated : 0.4300 truth : 0.5000 1-NN finds each test row's twin sitting in the training set at distance zero and returns its label. Accuracy 92% on data with no signal at all.
6. Summary: the sizes, side by side
Every number below was produced on data containing no signal whatsoever, where the only honest answer is 0.50. The optimism column is therefore exactly the size of the lie.
==============================================================================
EVERY LEAK ON THIS PAGE, MEASURED AGAINST THE TRUTH
==============================================================================
leak setup CV optimism
----------------------------------------------------------------------
none - proper pipeline 500 feats 0.4800 -0.0200
scaler fitted before split 500 feats 0.4800 -0.0200
feature selection before split 500 feats 0.6500 +0.1500
none - proper pipeline 10 feats 0.5400 +0.0400
target leakage (derived column) 10 feats + 1 0.8500 +0.3500
group leakage (patient split) 30 patients 0.9417 +0.4417
duplicate rows rows doubled 0.9200 +0.4200
----------------------------------------------------------------------
TRUTH (no signal exists) any 0.5000 +0.0000
The 'setup' column matters: each leak is compared against the honest
baseline for ITS OWN feature table, not across rows. The two baseline
rows show how much the estimate wanders on noise alone.
Note the ordering. The leak everyone knows about -- forgetting to put
the scaler in a pipeline -- is the smallest one here. The expensive
ones are structural: selecting features on all the data, a column
that encodes the answer, and rows that are not independent.
The checklist this page argues for
==============================================================================
1. Split first. Everything fitted -- scalers, imputers, encoders, feature selectors -- goes inside a Pipeline.
2. Ask of every feature: would this value exist, with this value, at the moment of prediction?
3. Ask what an independent unit is. If rows share a patient, session, or device, split on that unit.
4. If time matters, split on time. Never shuffle.
5. Deduplicate before splitting, and look for near-duplicates too.
6. Be suspicious of good news. Score features individually; a single column carrying the model is the signature of a leak.
Verified in this notebook
Every experiment runs on independent random features and shuffled balanced labels, so the true accuracy is exactly 0.50 and every excess is measured, not argued:
- Fitting a scaler before the split: small but real optimism
- Selecting 5 of 500 features before the split: large optimism, entirely manufactured
- A target-derived column defeats an otherwise perfect pipeline
- Random vs
GroupKFoldon 30 patients × 4 recordings - Shuffled vs forward-chaining CV on trended data
- 1-NN with duplicated rows recovers near-perfect accuracy from pure noise
Where this connects
[[cross-validation]] is the machinery these leaks corrupt — in particular the group and time-series splitters used here. [[evaluation-metrics]] is what you report once the estimate is honest.
References
- Leakage in Data Mining: Formulation, Detection, and Avoidance
- Leakage and the Reproducibility Crisis in ML-based Science