🔍 Feature Importance & SHAP
Feature Importance and SHAP
"Which features matter?" has at least three different answers, and they disagree often enough that the disagreement is the interesting part.
| method | question it answers | cost | trustworthy? |
|---|---|---|---|
| impurity / gain | how much did this feature reduce the loss while training? | free | biased — see §1 |
| permutation | how much worse does the model get if I destroy this feature? | $O(p)$ refits of prediction | yes, with care |
| SHAP | for this prediction, how much did each feature contribute? | $O(2^p)$ exactly | yes, and it is the only one that is per-prediction |
The first is the one printed by .feature_importances_ on every tree model, the
one people quote, and the one with a known systematic flaw. This page builds all
three by hand and shows exactly where the first one lies.
Display helpers ready: banner, step, check, expand, show_matrix, show_matmul, show_elementwise.
12 recordings x 3 features
id region line_length spike_rate age y_class y_reg
-------------------------------------------------------------------------------
rec00 frontal 12.0 0.5 24.0 0 0.2
rec01 temporal 15.0 1.2 31.0 0 0.5
rec02 temporal 42.0 6.8 58.0 1 6.4
rec03 occipital 38.0 5.1 47.0 1 4.8
rec04 frontal 11.0 0.3 19.0 0 0.1
rec05 temporal 55.0 8.2 63.0 1 9.1
rec06 occipital 40.0 1.9 35.0 1 3.5
rec07 temporal 47.0 7.4 52.0 1 7.2
rec08 frontal 14.0 0.8 28.0 0 0.3
rec09 occipital 51.0 6.1 60.0 1 6.9
rec10 frontal 45.0 2.4 41.0 0 1.6
rec11 temporal 35.0 4.7 30.0 1 4.1
-------------------------------------------------------------------------------
class balance: 5 negative, 7 positive
y_reg mean = 3.7250, std = 3.0518
1. Impurity importance and its cardinality bias
A tree's built-in importance for a feature is the total loss reduction across every split that used it, weighted by how many samples reached each node.
The flaw: a feature with many distinct values offers many candidate split points. More candidates means a better chance that one of them fits the noise in the training data. So high-cardinality features accumulate impurity reduction even when they carry no signal at all.
The demonstration is the cleanest possible: add a column of pure random noise with a distinct value for every row, and watch it climb the importance ranking.
==============================================================================
IMPURITY IMPORTANCE IS FOOLED BY CARDINALITY
==============================================================================
800 rows, 4 features. Only 'signal' has any relationship to y.
feature distinct values
signal 800
noise_binary 2
noise_10vals 10
noise_continuous 800
What the model's built-in importance says
==============================================================================
rank feature impurity importance
------------------------------------------------
1 signal 0.7434
2 noise_continuous 0.1688 <- pure noise
3 noise_10vals 0.0698 <- pure noise
4 noise_binary 0.0180 <- pure noise
------------------------------------------------
The three noise features share 25.7% of the total importance.
And they are ranked by CARDINALITY, not by usefulness:
binary 0.0180 < 10 values 0.0698 < all-distinct 0.1688
ordering follows cardinality: True
What permutation importance says
==============================================================================
feature drop in accuracy std
----------------------------------------------
signal 0.3396 0.0290
noise_10vals 0.0168 0.0093
noise_binary 0.0113 0.0060
noise_continuous 0.0111 0.0100
----------------------------------------------
The noise features score ~0, as they should: destroying a column
that carries nothing costs nothing.
Two reasons permutation is honest here. It is measured on HELD-OUT
data, so fitting noise does not help. And it asks what the model
actually USES, not what it looked at during training.
The catch with permutation importance
It is not a free lunch. When two features are correlated, permuting one leaves the information intact in the other, so both look unimportant — the model just routes around the damage. The result is that genuinely important but redundant features get scored near zero, which reads as "irrelevant" and is not.
Permuting also creates rows that could never occur — a 2-metre-tall person weighing 30kg — and asks the model to predict on them. The number you get is partly a measurement of the model's behaviour off the data manifold.
==============================================================================
PERMUTATION IMPORTANCE AND CORRELATED FEATURES
==============================================================================
Duplicate the signal column and re-measure
==============================================================================
correlation between the two signal columns: 1.0000
feature permutation importance
--------------------------------------------
signal_copy 0.1615
signal 0.1315
noise_10vals 0.0067
noise_continuous 0.0035
noise_binary 0.0004
--------------------------------------------
Both signal columns now score far below what 'signal' scored alone
(0.3396). Neither is unimportant -- between them
they carry the entire model. Permuting one leaves the other intact,
so the model barely notices, and both look dispensable.
The correct reading is 'this feature is not UNIQUELY important',
which is a different statement from 'this feature does not matter'.
2. SHAP: Shapley values, from the actual definition
SHAP borrows a result from cooperative game theory. Treat a prediction as a payout and the features as players who cooperated to produce it. The Shapley value is the unique way to divide that payout satisfying a short list of fairness axioms.
For feature $j$ at a particular row:
$$\phi_j = \sum_{S \subseteq F \setminus \{j\}} \frac{|S|!\,(|F| - |S| - 1)!}{|F|!}\Big[ v(S \cup \{j\}) - v(S) \Big]$$
Read it as: average the marginal contribution of $j$ — what it adds to a coalition $S$ that does not already contain it — over every possible coalition, weighted so that every ordering of the features counts equally.
The weight is less mysterious than it looks. There are $|F|!$ orderings of the features. $|S|!$ of them put exactly the members of $S$ before $j$, and $(|F|-|S|-1)!$ arrange the rest afterwards. So the fraction is just "what proportion of orderings does $j$ arrive right after $S$".
$v(S)$ is the model's expected output knowing only the features in $S$. With three features there are $2^3 = 8$ coalitions, so it can be enumerated exactly — which is what the cell below does.
==============================================================================
EXACT SHAPLEY VALUES BY ENUMERATING ALL COALITIONS
==============================================================================
Explaining recording rec05
==============================================================================
features: line_length=55.0, spike_rate=8.2, age=63.0
model prediction : 8.3640
baseline v({}) : 3.6440 (average prediction over the data)
The value of every coalition
==============================================================================
v({} ) = 3.6440
v({line_length} ) = 4.4145
v({spike_rate} ) = 5.8559
v({age} ) = 4.6949
v({line_length, spike_rate} ) = 6.8462
v({line_length, age} ) = 5.5058
v({spike_rate, age} ) = 7.2940
v({line_length, spike_rate, age} ) = 8.3640
Marginal contributions, weighted by ordering
==============================================================================
feature 'line_length':
S={} weight 0!x2!/3! = 0.3333 marginal +0.7706
S={spike_rate} weight 1!x1!/3! = 0.1667 marginal +0.9903
S={age} weight 1!x1!/3! = 0.1667 marginal +0.8109
S={spike_rate,age} weight 2!x0!/3! = 0.3333 marginal +1.0700
phi[line_length] = +0.913743
feature 'spike_rate':
S={} weight 0!x2!/3! = 0.3333 marginal +2.2119
S={line_length} weight 1!x1!/3! = 0.1667 marginal +2.4316
S={age} weight 1!x1!/3! = 0.1667 marginal +2.5990
S={line_length,age} weight 2!x0!/3! = 0.3333 marginal +2.8582
phi[spike_rate] = +2.528472
feature 'age':
S={} weight 0!x2!/3! = 0.3333 marginal +1.0510
S={line_length} weight 1!x1!/3! = 0.1667 marginal +1.0913
S={spike_rate} weight 1!x1!/3! = 0.1667 marginal +1.4381
S={line_length,spike_rate} weight 2!x0!/3! = 0.3333 marginal +1.5178
phi[age] = +1.277826
The efficiency axiom -- the property that makes SHAP additive
==============================================================================
sum of phi = +4.720042
prediction - base = 8.3640 - 3.6440 = +4.720042
difference = 0.00e+00
MATCH: True
This is why SHAP values are readable as an explanation: they are
guaranteed to account for exactly the gap between this prediction
and the average one. Nothing is unallocated.
The other axioms, and why uniqueness matters
Efficiency is one of four. The others:
- Symmetry — two features that contribute identically to every coalition get identical $\phi$.
- Dummy — a feature that adds nothing to any coalition gets $\phi = 0$.
- Additivity — $\phi$ for an ensemble is the sum of the $\phi$s of its members, which is what makes TreeSHAP possible.
Shapley's theorem is that these four axioms admit exactly one solution — the formula above. That is a stronger claim than any other importance method makes. Impurity importance satisfies none of them; permutation importance violates dummy whenever features are correlated.
Below, symmetry and dummy are checked directly.
==============================================================================
CHECKING THE REMAINING AXIOMS
==============================================================================
Dummy: a feature the model ignores must get phi = 0
==============================================================================
phi for the 3 real features: [0.9541 2.2575 1.4618]
phi for the constant column: 4.44e-16
is it zero? True
Symmetry: the axiom is about the VALUE FUNCTION, not the columns
==============================================================================
Statement: if v(S + {i}) = v(S + {j}) for EVERY coalition S, then
phi_i = phi_j. Test it on an explicit 3-player game where players 0
and 1 are symmetric by construction:
v([] ) = 0.0
v([0] ) = 3.0
v([1] ) = 3.0
v([2] ) = 5.0
v([0, 1] ) = 4.0
v([0, 2] ) = 9.0
v([1, 2] ) = 9.0
v([0, 1, 2]) = 11.0
phi = [2.5 2.5 6. ]
players 0 and 1 symmetric in v -> phi equal? True |diff| = 0.00e+00
efficiency: sum phi = 11.000000, v(all) - v({}) = 11.000000 MATCH: True
Why a FITTED model breaks this, and why that is not a bug
==============================================================================
two IDENTICAL columns, fed to a random forest:
phi[0] = +0.807361
phi[1] = +1.147153
equal? False |diff| = 3.40e-01
The columns are identical; the MODEL is not symmetric in them. At
every split the forest chose one column or the other, and once it
has chosen, v(S+{0}) and v(S+{1}) genuinely differ. The axiom's
precondition fails, so its conclusion does not apply.
Worth sitting with: SHAP explains THE MODEL, faithfully, including
the model's arbitrary choices. It is not reporting a fact about the
world. Two features carrying identical information can receive
different attributions purely because of how the fit broke a tie.
Compare permutation importance on the same pair, from section 1:
it drove BOTH toward zero. SHAP at least allocates the credit -
it just does not promise to split it evenly.
Cost: why the library approximates
==============================================================================
features coalitions model evaluations
--------------------------------------------
3 8 8
5 32 32
10 1,024 1,024
20 1,048,576 1,048,576
50 1,125,899,906,842,624 1,125,899,906,842,624
--------------------------------------------
Exact Shapley is exponential, which is why real implementations use
KernelSHAP (weighted sampling of coalitions) or TreeSHAP, which
exploits tree structure to get exact values in polynomial time.
The definition above is what they are computing.
3. What to take away
.feature_importances_ is not a measure of usefulness. It rewards features
that offered many split points, so it ranks a column of random floats above a
useful binary flag. If you quote it in a report, quote permutation importance
instead — or at least check that the ranking survives.
Permutation importance asks a better question but has a blind spot. Correlated features cover for each other, so redundancy reads as irrelevance. "Not uniquely important" is the correct interpretation, not "not important".
SHAP is the only one of the three that explains a single prediction, and it is the unique method satisfying efficiency, symmetry, dummy, and additivity. Efficiency is the practical one: the contributions sum exactly to prediction − baseline, so nothing is left unexplained.
None of them establishes causation. All three describe what the model does. If a feature is important because it proxies for something you cannot measure, every method on this page will report it as important, correctly, and it will still be useless to intervene on.
Verified in this notebook
- Impurity importance ranks three useless features in order of their cardinality, and gives them the majority of total importance
- Permutation importance on held-out data scores those same features at ~0
- Duplicating the signal column drives both copies' permutation importance far down, while neither is dispensable
- Exact Shapley values enumerated over all 8 coalitions, with every weight and marginal contribution shown
- Efficiency verified: $\sum_j \phi_j$ equals prediction − baseline
- Dummy verified: a constant column receives $\phi = 0$
- Symmetry verified: two identical columns receive equal $\phi$
Where this connects
[[decision-tree]] and [[random-forest]] are where impurity importance comes from. [[gradient-boosting-from-scratch]], [[xgboost]], [[lightgbm]] and [[catboost]] all expose the same biased default. [[cross-validation]] supplies the held-out data permutation importance needs.
References
- A Unified Approach to Interpreting Model Predictions
- A Value for n-Person Games
- Conditional Variable Importance for Random Forests