🔁 Cross-Validation & Model Selection
Cross-Validation and Model Selection
Every other walkthrough in this section answers how does this algorithm work. This one answers a different question: how do you know whether it works, and why the obvious way of checking is wrong often enough to matter.
The short version: a single train/test split gives you one number, and that number has a standard error nobody prints. Cross-validation gives you a distribution instead. And the moment you start choosing between models using that distribution, you introduce a new bias that needs a second layer of validation to remove.
Everything here is worked on 12 EEG recordings so the folds can be written out by hand, then checked against scikit-learn.
Display helpers ready: banner, step, check, expand, show_matrix, show_matmul, show_elementwise.
12 recordings x 3 features
id region line_length spike_rate age y_class y_reg
-------------------------------------------------------------------------------
rec00 frontal 12.0 0.5 24.0 0 0.2
rec01 temporal 15.0 1.2 31.0 0 0.5
rec02 temporal 42.0 6.8 58.0 1 6.4
rec03 occipital 38.0 5.1 47.0 1 4.8
rec04 frontal 11.0 0.3 19.0 0 0.1
rec05 temporal 55.0 8.2 63.0 1 9.1
rec06 occipital 40.0 1.9 35.0 1 3.5
rec07 temporal 47.0 7.4 52.0 1 7.2
rec08 frontal 14.0 0.8 28.0 0 0.3
rec09 occipital 51.0 6.1 60.0 1 6.9
rec10 frontal 45.0 2.4 41.0 0 1.6
rec11 temporal 35.0 4.7 30.0 1 4.1
-------------------------------------------------------------------------------
class balance: 5 negative, 7 positive
y_reg mean = 3.7250, std = 3.0518
1. The problem: one split gives one number, and the number moves
Hold out 25% of the data, score the model, report it. The trouble is that "25% of the data" is not one thing — it is a choice, and different choices give different answers.
Below, the same model on the same data, with nothing changed but the random seed that decides who lands in the test set.
==============================================================================
THE SAME MODEL, THE SAME DATA, DIFFERENT SPLITS
==============================================================================
Ten random 75/25 splits
==============================================================================
seed test recordings accuracy
------------------------------------------------
0 rec10, rec11, rec06 0.667
1 rec06, rec08, rec03 0.667
2 rec00, rec11, rec06 0.667
3 rec11, rec04, rec02 1.000
4 rec11, rec04, rec02 1.000
5 rec11, rec02, rec08 1.000
6 rec09, rec05, rec04 1.000
7 rec07, rec10, rec06 0.667
8 rec09, rec05, rec08 1.000
9 rec06, rec02, rec10 0.667
------------------------------------------------
mean 0.833
range 0.667 - 1.000
Ten defensible answers between 67% and 100%.
Reporting any single one of them is reporting a coin flip as a fact.
Worse: the split can change WHICH MODEL WINS
==============================================================================
seed tree logistic winner
----------------------------------------
0 0.750 0.750 tie
1 0.750 0.750 tie
2 0.750 0.500 tree
3 1.000 1.000 tie
4 0.750 0.750 tie
5 1.000 1.000 tie
6 1.000 1.000 tie
7 0.750 0.750 tie
8 1.000 1.000 tie
9 0.750 0.750 tie
----------------------------------------
tree wins 1x, logistic wins 0x, tied 9x
If you had run seed 0 and stopped, you would have shipped one model.
Seed 1 would have shipped the other. Nothing about the models changed.
2. k-fold, by hand
The fix is to stop throwing data away. Split the data into $k$ equal folds; train $k$ times, each time holding out a different fold; every observation is tested on exactly once, and every observation trains the model $k-1$ times.
$$\text{CV}_k = \frac{1}{k}\sum_{i=1}^{k} \text{score}\big(f^{(-i)},\ \text{fold}_i\big)$$
where $f^{(-i)}$ is the model trained on everything except fold $i$.
With 12 recordings and $k = 4$, each fold is exactly 3 recordings — small enough to write every fold out and check the arithmetic.
==============================================================================
4-FOLD CROSS-VALIDATION, WRITTEN OUT
==============================================================================
The folds
==============================================================================
fold 0: ['rec00', 'rec04', 'rec08'] classes [0, 0, 0]
fold 1: ['rec01', 'rec05', 'rec09'] classes [0, 1, 1]
fold 2: ['rec02', 'rec06', 'rec10'] classes [1, 1, 0]
fold 3: ['rec03', 'rec07', 'rec11'] classes [1, 1, 1]
Train k times, hold out one fold each time
==============================================================================
fold train on test on correct acc
----------------------------------------------------------------
0 9 rec00, rec04, rec08 3/3 1.000
1 9 rec01, rec05, rec09 3/3 1.000
2 9 rec02, rec06, rec10 2/3 0.667
3 9 rec03, rec07, rec11 1/3 0.333
----------------------------------------------------------------
CV_4 = (1.000 + 1.000 + 0.667 + 0.333) / 4 = 0.7500
Every recording was tested exactly once: 9/12 correct overall = 0.7500
(equal to the fold mean here only because the folds are the same size)
Check against scikit-learn
==============================================================================
by hand : [1. 1. 0.6667 0.3333] mean 0.7500
scikit-learn : [1. 1. 0.6667 0.3333] mean 0.7500
max|diff| = 0.00e+00 MATCH: True
3. The number nobody prints: the standard error
A CV score is a mean of $k$ numbers, so like any mean it has a spread. The usual estimate is
$$\text{SE} = \frac{s}{\sqrt{k}}, \qquad s = \text{standard deviation of the fold scores}$$
Reporting CV = 0.83 without it invites a comparison that the data cannot
support. Two models differing by 0.02 with a standard error of 0.09 are not
distinguishable — and this is the single most common way model comparisons go
wrong.
A caution the formula hides: the $k$ fold scores are not independent, because their training sets overlap heavily (any two share $k-2$ folds). So $s/\sqrt{k}$ understates the true uncertainty. It is a floor, not an estimate — treat a difference smaller than one SE as no difference at all.
==============================================================================
HOW UNCERTAIN IS A CV SCORE?
==============================================================================
Fold-to-fold spread on our 12 recordings
==============================================================================
fold scores : [1. 1. 0.6667 0.3333]
mean : 0.7500
std (ddof=1) : 0.3191
SE = s/sqrt(k) = 0.3191/sqrt(4) = 0.1596
report: 0.750 +/- 0.160
Why that matters when comparing models
==============================================================================
model CV mean SE fold scores
------------------------------------------------------------
tree depth 1 0.9167 0.0833 [1. 1. 0.667 1. ]
tree depth 2 0.7500 0.1596 [1. 1. 0.667 0.333]
tree depth 4 0.9167 0.0833 [1. 1. 0.667 1. ]
logistic 0.9167 0.0833 [1. 1. 0.667 1. ]
naive bayes 0.9167 0.0833 [1. 1. 0.667 1. ]
------------------------------------------------------------
best mean: 'tree depth 1' at 0.9167 (SE 0.0833)
Within one SE of the best -- i.e. NOT distinguishable from it:
tree depth 1 0.9167
tree depth 4 0.9167
logistic 0.9167
naive bayes 0.9167
With 12 recordings the data simply cannot separate these. Picking the
top row and claiming it is 'the best model' overstates what you know.
The 1-SE rule
==============================================================================
A standard response: among models within one SE of the best, choose
the SIMPLEST rather than the highest-scoring. It costs nothing in
measurable accuracy and buys robustness, because the ranking inside
that band is mostly noise.
4. Choosing the right splitter
Plain k-fold assumes every row is an independent draw from the same distribution. When that assumption fails — and it usually does — plain k-fold gives an answer that is confidently wrong. Each variant below fixes one specific violation.
| Splitter | Use when | What breaks without it |
|---|---|---|
| Stratified k-fold | classification, especially imbalanced | a fold with no positives makes some metrics undefined and all of them unstable |
| Leave-one-out | very small $n$ | nothing breaks, but the estimate has high variance and costs $n$ fits |
| Group k-fold | repeated measures — several rows per patient/subject/session | the same subject appears in train and test: you measure memorisation, not generalisation |
| TimeSeriesSplit | anything ordered in time | training on the future to predict the past |
4.1 Stratified
Plain k-fold on 12 recordings with 7 positives can easily produce a fold that is all one class. Stratified k-fold keeps each fold's class ratio close to the whole dataset's.
==============================================================================
STRATIFIED K-FOLD
==============================================================================
Plain k-fold: what the class balance does
==============================================================================
overall: 5 negative, 7 positive (58% positive)
fold test classes % positive
--------------------------------------
0 [0, 0, 1] 33%
1 [1, 1, 1] 100%
2 [0, 1, 1] 67%
3 [0, 1, 0] 33%
--------------------------------------
Stratified: each fold mirrors the whole
==============================================================================
fold test classes % positive
--------------------------------------
0 [0, 0, 1] 33%
1 [1, 1, 0] 67%
2 [0, 1, 1] 67%
3 [0, 1, 1] 67%
--------------------------------------
spread of fold positive-rate: plain vs stratified
plain : [0.333, 1.0, 0.667, 0.333] range 0.667
stratified : [0.333, 0.667, 0.667, 0.667] range 0.333
With rarer positives this stops being cosmetic: a fold with zero
positives makes recall undefined and AUC uncomputable.
Leave-one-out: k = n
==============================================================================
12 fits, each testing one recording
per-recording correct: [1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1]
LOO accuracy = 10/12 = 0.8333
compare 4-fold: 0.7500
LOO uses the most training data (n-1 rows) so it is nearly unbiased,
but each test set is a single row, so the estimate is high-variance
and it costs n fits. For n=12 that is fine; for n=100k it is not.
4.2 Group k-fold — the one that bites EEG work
Suppose the 12 recordings come from 6 patients, two recordings each. Two recordings from the same patient are not independent: they share anatomy, electrode placement, baseline rhythms.
Plain k-fold will happily put patient 3's first recording in the training set and their second in the test set. The model can then score well by recognising the patient rather than the pathology — and the reported accuracy is measuring something you will never have at deployment, when every patient is new.
Group k-fold keeps all rows from a group on the same side of every split.
==============================================================================
GROUP K-FOLD: WHEN ROWS ARE NOT INDEPENDENT
==============================================================================
The data has hidden structure
==============================================================================
id patient y
rec00 P0 0
rec01 P0 0
rec02 P1 1
rec03 P1 1
rec04 P2 0
rec05 P2 1
rec06 P3 1
rec07 P3 1
rec08 P4 0
rec09 P4 1
rec10 P5 0
rec11 P5 1
Plain k-fold splits patients across train and test
==============================================================================
fold 0: test patients [1, 2, 5] | ALSO IN TRAIN: [2, 5]
fold 1: test patients [0, 3] | ALSO IN TRAIN: []
fold 2: test patients [2, 4, 5] | ALSO IN TRAIN: [2, 5]
4 patient-level leaks across 3 folds.
GroupKFold keeps each patient whole
==============================================================================
fold 0: test patients [2, 5] | also in train: none
fold 1: test patients [1, 4] | also in train: none
fold 2: test patients [0, 3] | also in train: none
What the difference is worth, in accuracy
==============================================================================
plain 3-fold : [0.75 0.75 1. ] mean 0.8333
group 3-fold : [0.75 1. 0.75] mean 0.8333
optimism from ignoring groups: +0.0000
The group number is the one that predicts performance on a NEW
patient, which is the only question anyone actually asked.
4.3 Time series — shuffling is cheating
If rows are ordered in time, a random split trains on the future to predict the past. The model can learn from information that would not exist at prediction time, and the score is meaningless.
TimeSeriesSplit produces expanding windows: always train on a prefix, always
test on the block immediately after it. Note the consequence — the training set
size differs between folds, so the fold scores are not identically distributed,
and averaging them is a rougher approximation than in plain k-fold.
==============================================================================
TIME-SERIES SPLITTING
==============================================================================
Expanding-window folds
==============================================================================
fold train (in time order) test
----------------------------------------------------------
0 [0, 1, 2, 3] [4, 5]
1 [0, 1, 2, 3, 4, 5] [6, 7]
2 [0, 1, 2, 3, 4, 5, 6, 7] [8, 9]
3 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] [10, 11]
----------------------------------------------------------
Train indices are always BEFORE test indices. No fold ever sees the
future. Note the training set grows: fold 0 trains on 4 rows, fold 3
on 10 -- so the folds are not measuring quite the same thing.
What a random split would do here
==============================================================================
fold 0: test [4, 6, 11] -> 2 test rows precede some training row
fold 1: test [2, 8, 10] -> 3 test rows precede some training row
fold 2: test [1, 7, 9] -> 3 test rows precede some training row
fold 3: test [0, 3, 5] -> 3 test rows precede some training row
11 causality violations. Every one is the model being handed
information that will not exist when it runs for real.
5. Nested CV — the bias you create by choosing
Here is the subtle one, and the reason this page exists.
Section 3 used cross-validation to compare five models and pick the best. But the winning score was the maximum over five noisy estimates — and the maximum of noisy numbers is biased upward. Some of that model's advantage is real skill, and some is simply that it got the friendlier folds.
Report that maximum as your expected performance and you are reporting a number that includes the luck. The more candidates you try, the worse it gets.
Nested cross-validation separates the two jobs:
- an inner loop searches hyperparameters, using only the training part of each outer fold
- an outer loop scores the whole procedure — "tune, then fit" — on data the inner loop never saw
The outer score answers the honest question: if I run my entire model-selection process on new data, how well will the result do?
The demonstration below is the important one on this page. It uses pure noise — features with no relationship to the target whatsoever, where the true accuracy of any model is chance. A flat (non-nested) search still reports well above chance. Nested CV does not.
==============================================================================
SELECTION BIAS: FLAT SEARCH vs NESTED CV, ON PURE NOISE
==============================================================================
60 samples, 200 features, all random.
class balance: [30, 30] -- deliberately exact, so
the majority-class baseline is 0.50 and cannot be mistaken for skill.
There is no relationship to learn. Honest accuracy = 0.50.
Flat search: score all 40 candidates on the same folds, report the best
==============================================================================
candidate CV scores: min 0.2833 mean 0.4800 max 0.6167 (std 0.0789)
best candidate : #24
best CV : 0.6167 <- what a flat search reports
Note the mean of all candidates is 0.4800 -- close to the
truth. It is only the MAXIMUM that is inflated.
Nested CV: pick inside, score outside
==============================================================================
outer fold 0: inner picked candidate # 8 (inner CV 0.6644) -> outer score 0.4167
outer fold 1: inner picked candidate #25 (inner CV 0.6467) -> outer score 0.3333
outer fold 2: inner picked candidate # 5 (inner CV 0.6622) -> outer score 0.5000
outer fold 3: inner picked candidate #24 (inner CV 0.7133) -> outer score 0.4167
outer fold 4: inner picked candidate #21 (inner CV 0.6689) -> outer score 0.3333
nested CV = 0.4000 <- the honest number
The gap
==============================================================================
flat (chose and reported on the same folds) 0.6167
nested (choice kept inside the inner loop) 0.4000
truth (labels are random, classes balanced) 0.5000
--------------------------------------------------------
optimism of the flat search +0.1167
optimism of nested CV -0.1000
The flat search reports 62% on data containing no signal.
The arithmetic is not wrong -- that really is the best of 40
scores. But it is the MAXIMUM of 40 noisy numbers, and the
maximum of noise is not zero.
The bias grows with the number of candidates
==============================================================================
Given the 40 candidate scores above: if you had tried only n of
them, picked at random, what would your reported best be?
Averaged over 3000 random draws:
n tried E[best CV] optimism
--------------------------------
1 0.4801 -0.0199
2 0.5239 +0.0239
5 0.5668 +0.0668
10 0.5863 +0.0863
20 0.6032 +0.1032
40 0.6167 +0.1167
--------------------------------
monotonically increasing: True
Trying one candidate is roughly honest. Trying forty and reporting
the winner is not. Nothing about the data changed between those
rows -- only how many chances you gave yourself.
This is why 'we tried 500 configurations and got 0.94' should raise
an eyebrow, not confidence.
Being fair to nested CV: unbiased is not the same as exact
==============================================================================
Nested CV came out at 0.4000 just now -- BELOW the true 0.50.
That is not a failure, it is variance: 5 outer folds of 12 rows each.
The claim is that nested CV has no systematic upward pull, not that
it hits 0.50 on one run. Repeating the whole experiment on 12 fresh
noise datasets:
dataset flat nested
----------------------------
0 0.6167 0.5000
1 0.6667 0.5000
2 0.6500 0.3833
3 0.6500 0.5667
4 0.6833 0.4500
5 0.6833 0.5000
6 0.7333 0.5667
7 0.6500 0.5000
8 0.7000 0.3667
9 0.6333 0.4500
10 0.6500 0.5500
11 0.6333 0.3333
----------------------------
mean 0.6625 0.4722
truth 0.5000 0.5000
flat search : 0.6625 (+0.1625) above 0.50 on 12/12 datasets
nested CV : 0.4722 (-0.0278) above 0.50 on 3/12 datasets
Averaged over repetitions the flat search sits clearly above chance
and nested CV sits close to it. That is the difference between a
biased estimator and a noisy one -- more data fixes noise, but no
amount of data fixes the bias in a flat search.
6. What to take away
A single split is a sample, not a measurement. It has a standard error, and on small data that error is large enough to change which model appears to win.
Cross-validation gives you a distribution. Use the spread. A difference smaller than one standard error is not a difference — and because the folds share training data, that standard error is itself an optimistic floor.
Match the splitter to the dependence in your data. Independent rows: k-fold. Classification: stratify. Repeated measures per subject: group. Ordered in time: expanding window. Getting this wrong does not produce a noisy answer, it produces a confident wrong one — the group-vs-plain gap on the EEG data was entirely manufactured by patients appearing on both sides of the split.
Selecting is measuring. Every hyperparameter you compare on a set of folds spends some of those folds' independence. Tune inside an inner loop, report from an outer one. The noise experiment is the proof: 200 random features, random labels, and a flat grid search still reported well above chance.
Verified in this notebook
- Hand-computed 4-fold scores match
cross_val_scoreexactly - Every recording tested exactly once; fold mean equals pooled accuracy for equal folds
- Stratified folds hold the class ratio far tighter than plain folds
GroupKFoldremoves the patient overlap that plain k-fold creates, and the accuracy gap that overlap causedTimeSeriesSplitnever places a training row after a test row- On data with no signal at all, flat search reports well above 0.50 while nested CV does not
Where this connects
[[data-leakage]] is the other half of this story — cross-validation only protects you if the folds are honest, and leakage is what makes them dishonest. [[hyperparameter-search]] is the inner loop of section 5. [[probability-calibration]] needs out-of-fold predictions, which is the machinery built here.
References
- A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
- On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation