🧱 Stacking & Blending
Stacking and Blending
There are three ways to combine models, and this section has covered two of them:
| family | how it combines | example |
|---|---|---|
| bagging | average many models trained on resampled data | [[random-forest]] |
| boosting | fit each model to what the previous ones got wrong | [[gradient-boosting-from-scratch]] |
| stacking | train a model to combine the predictions of other models | this page |
Bagging reduces variance, boosting reduces bias, and stacking does something different from both: it learns how much to trust each model, and it can learn that the answer varies.
The whole method rests on one requirement, and getting it wrong silently ruins everything: the combiner must be trained on predictions the base models made for rows they did not see during training. That means out-of-fold predictions, which is why this page sits next to [[cross-validation]] and [[data-leakage]].
Display helpers ready: banner, step, check, expand, show_matrix, show_matmul, show_elementwise.
==============================================================================
FOUR BASE MODELS THAT DISAGREE
==============================================================================
2000 rows, 6 features, class balance [980, 1020]
signal = a linear term in features 0-2, PLUS sign(x3 * x4)
the second term is invisible to any linear model
Each on its own
==============================================================================
model CV accuracy
--------------------------
logistic 0.6046
forest 0.9046
nn1 0.7646
nbayes 0.6015
--------------------------
Do they make the SAME mistakes?
==============================================================================
pairwise agreement on predicted label:
logistic forest nn1 nbayes
logistic 1.000 0.651 0.588 0.952
forest 0.651 1.000 0.771 0.649
nn1 0.588 0.771 1.000 0.589
nbayes 0.952 0.649 0.589 1.000
rows at least one model gets wrong : 713 (54.8%)
rows ALL FOUR get wrong : 45 (3.5%)
The gap between those two numbers is the entire opportunity. On
668 rows, somebody in the ensemble already knew the answer.
Combining cannot help on the 45 rows nobody gets right.
1. The out-of-fold construction
The combiner (the meta-learner) needs a training set whose features are the base models' predictions. The question is where those predictions come from.
The wrong way, and it is tempting: fit each base model on the training set, predict on that same training set, and train the meta-learner on the result. The base models have already seen those rows. An overfitted model looks perfect on its own training data, so the meta-learner concludes it should trust the overfitted model completely — and that model is the one that generalises worst.
The right way — out-of-fold predictions:
- Split the training data into $k$ folds.
- For each fold $i$: train each base model on the other $k-1$ folds, predict fold $i$.
- Every row now has a prediction from a model that never saw it. Stack those columns into a meta-feature matrix $Z$.
- Train the meta-learner on $(Z, y)$.
- Refit each base model on all the training data, for use at prediction time.
The cost is $k+1$ fits per base model. The demonstration below shows why paying it is not optional.
==============================================================================
OUT-OF-FOLD PREDICTIONS, AND WHAT HAPPENS WITHOUT THEM
==============================================================================
Build Z the right way: cross_val_predict gives out-of-fold probabilities
==============================================================================
Z shape (1300, 4) -- one column per base model, one row per training row
row logistic forest nn1 nbayes y
0 0.2475 0.5667 0.0000 0.3190 0
1 0.7201 0.8833 1.0000 0.7237 1
2 0.3590 0.5000 1.0000 0.3551 1
3 0.3771 0.6900 1.0000 0.3797 1
4 0.6261 0.2700 0.0000 0.6333 0
5 0.3620 0.2067 0.0000 0.3890 0
Build Z the WRONG way: in-sample predictions
==============================================================================
How well does each column predict y, in-sample vs out-of-fold?
model in-sample acc out-of-fold acc
--------------------------------------------
logistic 0.6092 0.6046
forest 1.0000 0.9046
nn1 1.0000 0.7646
nbayes 0.6085 0.6015
--------------------------------------------
Two models score 1.0000 in-sample. Neither is perfect -- both have
memorised the training rows. Look at what happens to 'nn1' when it
is asked about rows it has NOT seen: 0.7646.
A meta-learner shown the left column has no way to know that.
What the meta-learner learns from each version
==============================================================================
model weight (leaky Z) weight (honest Z)
--------------------------------------------------
logistic 0.2917 -0.4883
forest 4.4258 10.1849
nn1 5.9692 0.6994
nbayes 0.2634 -1.2488
--------------------------------------------------
leaky Z puts the most weight on 'nn1'
honest Z puts the most weight on 'forest'
'nn1' has 100% in-sample accuracy and 77.0% on the test set.
'forest' has 90.0% on the test set.
The leak did not add noise -- it pointed the meta-learner at the
wrong model, confidently.
The cost, measured on the held-out test set
==============================================================================
base model test accuracy
----------------------------------------
forest 0.9000
nn1 0.7700
logistic 0.5929
nbayes 0.5900
----------------------------------------
stacking, leaky Z 0.7700
stacking, out-of-fold 0.9043
----------------------------------------
out-of-fold vs leaky Z : +0.1343
out-of-fold vs best single model : +0.0043 (best was 'forest')
2. What the meta-learner actually learns
With a logistic meta-learner the fitted coefficients are readable: they are how much each base model is trusted, on the log-odds scale. Two things are worth noticing.
The weights are not proportional to accuracy. A model that is slightly worse overall but wrong in different places can receive more weight than a slightly better one that duplicates an existing member. Stacking rewards complementarity, not just quality — which is why an ensemble of four mediocre, diverse models often beats four good, similar ones.
Weights can be negative. A model that is reliably wrong in a particular region is informative; the meta-learner can subtract it. This is something no simple average can do.
==============================================================================
WEIGHTS, DIVERSITY, AND THE SIMPLE-AVERAGE BASELINE
==============================================================================
Fitted trust vs standalone accuracy
==============================================================================
model own CV acc meta weight corr with others
--------------------------------------------------------
logistic 0.6046 -0.4883 0.6061
forest 0.9046 10.1849 0.5770
nn1 0.7646 0.6994 0.3861
nbayes 0.6015 -1.2488 0.6086
--------------------------------------------------------
Note the NEGATIVE weights: logistic, nbayes.
A simple average cannot do that. A model that is reliably wrong in
a particular region is informative, and the meta-learner subtracts
it rather than being dragged along by it.
Stacking vs simple averaging
==============================================================================
method test accuracy test log loss
----------------------------------------------------------
best single model 0.9000
simple average of the 4 0.8043 0.4825
stacked (logistic meta) 0.9043 0.2566
----------------------------------------------------------
Look at the middle row. The simple average scores 0.8043 -- WORSE than
simply using 'forest' alone (0.9000). Two of the four base
models are near-useless on this problem, and averaging gives them
the same vote as the good ones.
Stacking's reliable win is exactly there: over the average, +0.1000.
Its win over the single best model is much smaller (+0.0043), and
that is the honest general picture -- stacking usually buys a
percent or two over your best model, which matters in a competition
and rarely justifies four models in production.
Check against scikit-learn's StackingClassifier
==============================================================================
by hand : 0.9043
scikit-learn : 0.9043
difference : 0.0000
Small differences are expected: StackingClassifier passes all 2
probability columns per estimator by default and refits the bases
on the full training set slightly differently. The construction is
the same one built above.
3. Blending, and the failure mode of both
Blending is the cheap variant: instead of $k$-fold out-of-fold predictions, hold out a single validation slice (say 20%), train the bases on the rest, and train the meta-learner on that one slice.
- cheaper — one fit per base model instead of $k+1$
- simpler to get right — there is no fold bookkeeping to botch
- but the meta-learner sees far less data, and the base models never train on the holdout
Use blending when base models are expensive; stacking when data is scarce.
The failure mode both share is the one from [[cross-validation]]: the meta-learner is itself a model selected using the training folds. Adding base models until the stacked CV score peaks is a search, and the winner's score is biased upward for exactly the reasons in that page. A stacked ensemble tuned this way needs an outer loop to be reported honestly — and stacking makes the problem worse than usual, because the number of things being compared is large and the gains being chased are small.
==============================================================================
BLENDING, AND WHY A STACKED SCORE NEEDS AN OUTER LOOP
==============================================================================
Blending: one holdout slice instead of k folds
==============================================================================
meta-learner trained on 325 rows (blending) vs 1300 (stacking)
base model fits: 4 (blending) vs 24 (stacking)
blended test accuracy : 0.8900
stacked test accuracy : 0.9043
Selection bias: adding members until the score peaks
==============================================================================
Trying every non-empty subset of the 4 base models.
The out-of-fold columns Z are computed once and reused, so each
subset only needs the meta-learner refitted -- the base models do
not change with the subset.
subset CV score test
--------------------------------------------------------
logistic+forest+nbayes 0.9115 0.9100
forest+nbayes 0.9108 0.9100
logistic+forest 0.9092 0.9114
forest+nn1+nbayes 0.9054 0.9029
forest 0.9046 0.9000
--------------------------------------------------------
chosen by CV : logistic+forest+nbayes (CV 0.9115, test 0.9100)
best on test : logistic+forest (CV 0.9092, test 0.9114)
same subset? False
cost of choosing by CV instead of an oracle: +0.0014 test accuracy
15 subsets were compared, so the winning CV score is the
maximum of 15 noisy numbers. Two consequences, and the
second is the one people miss:
1. that maximum is biased upward as an estimate of performance
2. the ARGMAX is unreliable too -- the subset CV picks is often
not the one that does best on new data, as above.
Both are the same phenomenon from [[cross-validation]]: with 15 candidates
and differences this small, the ranking is mostly noise. Report a
stacked ensemble from an outer loop, or do not report a number.
4. What to take away
Stacking learns trust; averaging assumes it. The meta-learner discovers how much each base model is worth, and can assign negative weight to one that is usefully wrong. That flexibility is what it buys over a simple average — and a simple average is a genuinely strong baseline that costs nothing, so make the comparison before adding the complexity.
Out-of-fold predictions are the whole method. Train the meta-learner on in-sample predictions and it will learn to trust whichever base model memorised best. This is [[data-leakage]] with extra steps, and it is invisible unless you check.
Diversity beats individual quality. The opportunity is the gap between "at least one model is wrong" and "all models are wrong". Adding a fifth model that agrees with an existing one widens neither.
And the stacked score needs an outer loop. Choosing which models to include, by comparing cross-validated stacked scores, is a search — with the same upward bias as any other search.
Verified in this notebook
- Base models agree on most rows but the "at least one wrong" and "all wrong" counts differ substantially, which is the room stacking has to work in
- In-sample meta-features give 1-NN — which memorises perfectly and generalises at 77% — the highest meta weight, and the ensemble collapses to exactly 1-NN's test accuracy. Out-of-fold meta-features pick the forest instead, worth +0.13 test accuracy
- The meta-learner assigns negative weights, which a simple average cannot
- The simple average scores below the best single model, because two of the four bases are near-useless and averaging gives them equal votes — this is the gap stacking reliably closes
- Hand-built stacking lands close to
StackingClassifier, with the remaining difference explained rather than hidden - All 15 subsets compared: the CV-selected subset is not the test-best one
Where this connects
[[cross-validation]] builds the out-of-fold machinery this depends on. [[data-leakage]] is what in-sample meta-features are an instance of. [[random-forest]] and [[gradient-boosting-from-scratch]] are the other two ways to combine models. [[probability-calibration]] matters here too — a badly-calibrated base model contributes a distorted meta-feature.
References
- Stacked Generalization
- Stacked Regressions