🧿 GLM-5 & GLM-5.3-Flash
GLM-5 and GLM-5.3-Flash
The third frontier architecture in this section, after [[kimi-k3]] and [[deepseek-v4]] — and the most useful one to read third, because it shares mechanisms with both and picks different trade-offs.
Two sources, and they are not equal. The GLM-5 technical report is a real paper with ablations, and everything in sections 1–5 below comes from reading it. GLM-5.3-Flash, the model you may have arrived here from, has only a product documentation page — no paper — so its specifics are flagged as such wherever they appear. That distinction is kept visible throughout rather than blended away.
| GLM-4.5 | GLM-5 | GLM-5.3-Flash | |
|---|---|---|---|
| total / active params | 355B / 32B | 744B / 40B | 320B / 18B (docs) |
| layers | — | 80 | 45 (docs) |
| experts | — | 256 | — |
| training tokens | — | 28.5T | 30T multimodal (docs) |
| attention | — | MLA → DSA | hybrid sparse + linear (docs) |
What makes GLM-5 worth studying is that the report shows its failures. It publishes an ablation of five efficient-attention schemes, one of which is a catastrophe, and that table explains the design better than the design does.
Display helpers ready: banner, step, check, expand, show_matrix, show_matmul, show_elementwise.
==============================================================================
THE ABLATION THAT EXPLAINS THE ARCHITECTURE
==============================================================================
Every variant, measured against full attention at 128K
==============================================================================
variant RULER MRCR HELMET-ICL RepoQA mean drop
----------------------------------------------------------------------------------
full attention 75.28 35.39 77.36 65.83 --
SWA Interleave 44.93 28.83 63.52 39.33 -19.31
SWA Pattern 69.59 33.58 74.60 51.17 -6.23
GDN 64.00 30.22 74.84 56.17 -7.16
SimpleGDN 67.03 31.27 81.84 58.50 -3.80
----------------------------------------------------------------------------------
Ranked by average damage
==============================================================================
SWA Interleave -19.31 ###############################
GDN -7.16 ###########
SWA Pattern -6.23 ##########
SimpleGDN -3.80 ######
The spread is the finding. Same baseline, same 190B tokens of
adaptation, same 1:1 layer ratio -- and the worst scheme loses
19.3 points on average while the best loses 19.3.
Why fixed sliding windows fail, and search fixes them
==============================================================================
Both SWA rows use the SAME mechanism and the SAME budget:
half the layers windowed at 4096 tokens, half full attention.
The only difference is WHICH layers get windowed.
context full attn interleaved searched
------------------------------------------------
4K 95.19 94.87 95.78
8K 93.67 54.02 92.54
16K 92.01 25.89 88.92
32K 91.09 12.61 82.52
64K 85.35 8.32 70.23
128K 75.28 6.51 53.95
------------------------------------------------
Interleaved SWA collapses to 6.51 at 128K -- worse than
guessing on a retrieval benchmark. The searched pattern, same layer
budget, holds 53.95.
The pattern the beam search found:
SFSSFFSSSFFFFSSFSFFFFFFSFSFSSFSSFSFSSFSSS
(S = sliding window, F = full attention, 40 layers)
S: 21 layers, F: 20 layers
Nothing about it is periodic. That is the point -- the assumption
behind interleaving is that layers are interchangeable, and they
are not. Note also the search ran only at 16K yet generalised to
128K.
1. Why GLM-5 chose DSA over all of them
The paper's conclusion from that table is worth quoting in substance: every scheme above discards information and pays for it on fine-grained retrieval, even with half the layers kept dense. Sliding windows throw away everything outside the window. Linear attention compresses history into a fixed-size state, so old detail is gone whether you needed it or not.
DSA is different in kind. Its lightning indexer scores every past token cheaply and keeps the top-$k$. Nothing is structurally excluded — a token from a million positions back is retrievable if the indexer ranks it highly. The paper calls this lossless by construction, and it is why DSA can be applied to all layers while the others need a 1:1 mix with full attention.
That mechanism is worked through in detail in [[deepseek-v4]] §3 — GLM-5 adopts it rather than inventing it. What GLM-5 contributes is how to get there cheaply.
==============================================================================
RETROFITTING SPARSE ATTENTION INSTEAD OF PRE-TRAINING IT
==============================================================================
The two-stage recipe, and its cost
==============================================================================
Stage 1, warm-up: train ONLY the indexer, base model frozen
1,000 steps x 14 sequences x 202,752 tokens, lr 5e-3
= 2.84B tokens
Stage 2, sparse adaptation: co-train model and indexer
= 20B tokens
GLM-5 total : 22.8B tokens
DeepSeek-V3.2 : 943.7B tokens
ratio : 41x cheaper
Same mechanism, same reported end quality. The saving is entirely
in not re-learning what the dense model already knew.
The GLM-4.7-Flash check: how much does the warm-up alone buy?
==============================================================================
A separate small-scale test. Two stages: (i) train ONLY the indexer
for 1,000 steps at batch size 16 with the base model frozen, then
(ii) co-train model and indexer on 150B tokens.
Only the figures the paper states exactly are shown here -- the
128K pair, and the deltas after full joint training:
stage RULER @128K
--------------------------------------------------
dense baseline 79.21
indexer warm-up only 71.35
--------------------------------------------------
drop from freezing the base model -7.86
After the 150B-token joint phase, relative to the dense baseline:
context delta
----------------------
16K +0.86
32K +0.49
64K +1.72
128K -0.35
----------------------
mean delta over those four: +0.68
So the indexer alone, with every base weight frozen, already holds
most of the quality, and the damage is concentrated at the longest
context. Joint training then closes it -- and IMPROVES on the dense
baseline at 16K, 32K and 64K, leaving only a 0.35 deficit at 128K.
That is the argument for retrofitting: the model does not need to
relearn language, only how to use the indexer.
2. Muon Split — orthogonalise each head separately
This is GLM-5's most interesting contribution and it connects directly to [[deepseek-v4]] §7, where Muon's Newton–Schulz orthogonalisation was worked through.
Recall what Muon does: replace the gradient $M = U\Sigma V^T$ with $UV^T$, so every direction gets an equally-sized step. The question GLM-5 asks is what counts as one matrix.
The standard recipe orthogonalises the whole up-projection $W^{UQ}$ at once. But that matrix is a stack of per-head blocks, and orthogonalising the stack couples them: the shared $\Sigma$ is dominated by whichever head had the largest gradient, so a head with small gradients gets a step scaled by another head's magnitude.
Muon Split slices the matrix by head and orthogonalises each block independently, letting each head's weights update at its own scale. The effect in the paper's ablation is not subtle — it is what makes MLA competitive with GQA-8 at all:
| variant | MMLU | C-Eval | RACE | HumanEval |
|---|---|---|---|---|
| GQA-8 | 61.2 | 60.0 | 79.6 | 38.5 |
| MLA | 61.5 | 59.7 | 77.8 | 33.5 |
| MLA + Muon Split | 62.5 | 62.1 | 79.9 | 36.7 |
| MLA-256 + Muon Split | 62.0 | 59.9 | 79.6 | 36.6 |
There is a second consequence the paper notes: with Muon Split, attention logits stay stable through pre-training without any clipping. Kimi needed QK-Clip; [[deepseek-v4]] avoided it with QK RMSNorm; GLM-5 gets it from the optimiser. Three papers, three routes around the same failure.
==============================================================================
MUON SPLIT: WHOLE-MATRIX vs PER-HEAD ORTHOGONALISATION
==============================================================================
A gradient whose heads have very different magnitudes
==============================================================================
W is (32 x 16) = 4 head blocks of (8 x 16)
head ||block||_F relative
--------------------------------
0 35.5606 1.0000
1 10.4636 0.2942
2 3.3350 0.0938
3 0.5867 0.0165
--------------------------------
head 3's gradient is 61x smaller than head 0's.
Standard Muon: orthogonalise the whole stack
==============================================================================
head ||update block||_F share of total
--------------------------------------------
0 2.7964 0.4887
1 2.3533 0.3461
2 1.6020 0.1604
3 0.2752 0.0047
--------------------------------------------
spread across heads: 10.16x
The head with the smallest gradient still receives the smallest
update -- the whole point of orthogonalisation was undone by
sharing one set of singular values across all four heads.
Muon Split: orthogonalise each head block on its own
==============================================================================
head ||update block||_F share of total
--------------------------------------------
0 2.8284 0.2500
1 2.8284 0.2500
2 2.8284 0.2500
3 2.8284 0.2500
--------------------------------------------
spread across heads: 1.0000x
every block now has 8 singular values equal to 1, so each
contributes sqrt(8) = 2.8284 to the norm regardless of
how large its gradient was.
Check the singular values directly
==============================================================================
head 3 (the weakest gradient), singular values of its update block:
whole-matrix Muon : [0.1882 0.1412 0.1082 0.0791 0.0316 0.0287 0.0209 0.0114]
Muon Split : [1. 1. 1. 1. 1. 1. 1. 1.]
all ones under Split: True
This is what 'projection weights for different attention heads
update at different scales' means, and why splitting fixes it.
3. MLA-256 — moving cost from decoding to training
MLA compresses keys and values into a latent, which is why its cache is small. The cost lands somewhere else: during decoding, MLA's dot product runs in the latent dimension — 576 for DeepSeek-V3 — against GQA's 128. Decoding is memory-bandwidth-bound and token-by-token, so that is exactly the wrong place to be expensive.
GLM-5's fix: raise the head dimension from 192 to 256 and use fewer heads. Parameter count and training compute both scale with $n_h \times d_{\text{head}}$, so holding that product fixed leaves training untouched. Decoding cost scales with $n_h$ alone — so fewer, fatter heads is strictly cheaper to decode.
The paper says heads decrease "by 1/3". Worth checking what the product constraint actually requires, because the two do not quite agree — the notebook computes both readings rather than picking one.
==============================================================================
THE HEAD-SHAPE TRADE
==============================================================================
head dimension 192 -> 256 (x1.3333)
Parameters and training FLOPs scale with n_h x d_head.
Decoding cost scales with n_h (one latent dot product per head).
reading n_h n_h x d vs old decode
------------------------------------------------------------------------
original 128 24,576 1.0000 1.0000
heads x 3/4 (product constant) 96 24,576 1.0000 0.7500
heads x 2/3 ('decrease by 1/3') 85 21,760 0.8854 0.6641
------------------------------------------------------------------------
Only the x3/4 reading holds parameters exactly constant, which is
what the paper claims. The x2/3 reading would shrink the model by
11.5%. Either way decoding gets cheaper by 25-33%,
which is the direction that matters.
Where the decoding cost actually goes
==============================================================================
per generated token, per layer, dot-product work:
MLA (DeepSeek-V3 shape): 128 heads x 576 dims = 73,728
MLA-256 (heads x 3/4): 96 heads x 576 dims = 55,296
GQA-8 for reference: 8 heads x 128 dims = 1,024
MLA's cache is small and its decode arithmetic is not. GLM-5 keeps
the cache saving and buys back 25% of the arithmetic.
The paper's justification for why this was available at all: the
head count in DeepSeek-V3 was tuned to the roofline of one specific
accelerator (H800), and is not optimal on different hardware.
4. Multi-token prediction with shared parameters
MTP predicts several tokens ahead. It improves the base model, and the extra heads double as a draft model for speculative decoding: propose $n$ tokens cheaply, verify them in one pass, keep the prefix that matches.
The tension: predicting $n$ tokens ahead normally needs $n$ MTP layers, so memory grows linearly in speculative depth. DeepSeek-V3 trains one layer and predicts two tokens at inference — cheap, but the second token is produced by a layer never trained for that position, and acceptance suffers.
GLM-5 shares one set of parameters across three MTP layers. Memory matches the single-layer cost; training matches the inference-time use.
The payoff is the accept length — how many drafted tokens survive verification on average. GLM-5: 2.76 against DeepSeek-V3.2's 2.55, at the same speculative depth of 4. Small-looking, and worth converting into what it means.
==============================================================================
SPECULATIVE DECODING: WHAT ACCEPT LENGTH BUYS
==============================================================================
Accept length -> speedup
==============================================================================
With accept length L, one verification pass emits L tokens.
Speedup over plain decoding is L, minus the drafting overhead.
model accept length raw speedup at 20% overhead
--------------------------------------------------------------------
DeepSeek-V3.2 2.55 2.55x 2.12x
GLM-5 2.76 2.76x 2.30x
--------------------------------------------------------------------
relative gain: +8.2% more tokens per verification pass
Where accept length comes from
==============================================================================
If each drafted token is accepted independently with probability p,
and drafting depth is D, the expected accepted prefix is
L = sum_{i=1..D} p^i (plus 1 for the always-correct token)
p expected prefix +1 verified
--------------------------------------
0.50 0.9375 1.9375
0.60 1.3056 2.3056
0.70 1.7731 2.7731
0.80 2.3616 3.3616
0.90 3.0951 4.0951
--------------------------------------
Inverting for the reported numbers at depth 4:
DeepSeek-V3.2 L=2.55 -> per-token acceptance p = 0.6552
GLM-5 L=2.76 -> per-token acceptance p = 0.6975
difference: +0.0423 per token
Verify by substituting back:
DeepSeek: 1 + sum p^i = 2.5500 target 2.55 MATCH: True
GLM-5: 1 + sum p^i = 2.7600 target 2.76 MATCH: True
A +0.042 improvement in per-token acceptance compounds over the
draft into +8.2% throughput. That is the return on training the
MTP layers in the configuration they are actually used in.
5. GLM-5.3-Flash — what the documentation claims
Everything above is from the technical report. GLM-5.3-Flash has no paper, so this section is a reading of the product documentation and should be treated accordingly.
The headline: 320B total / 18B active across 45 layers, 1M context, natively multimodal over images, video and files, trained on a 30T-token multimodal corpus. Two mechanisms are named:
- IndexShare — one indexer reused across every four sparse-attention layers, rather than one per layer. (Not in the paper; from Z.ai material.)
- IndexPool — four indexer key vectors compressed into one by weighted pooling.
Both attack the same overhead. DSA's indexer is cheap per token but it runs in every layer over the whole context, and at a million tokens that stops being negligible. The claimed effect is 3.01× less attention computation and 4.44× smaller KV cache relative to GLM-5.3.
It also uses Manifold-Constrained Hyper-Connections — the same mHC worked through in [[deepseek-v4]] §2, Sinkhorn projection onto the Birkhoff polytope and all. Two independent frontier labs adopting the same residual-stream mechanism within months is worth noticing.
The arithmetic below reconstructs where savings of that size could come from. It is a plausibility check on stated numbers, not a reproduction — the docs give no configuration to reproduce.
==============================================================================
RECONSTRUCTING THE GLM-5.3-FLASH EFFICIENCY CLAIMS
==============================================================================
Source: Z.ai documentation. No paper, no configuration published.
Treat the reconstruction as an order-of-magnitude check.
IndexShare: one indexer per four layers instead of one per layer
==============================================================================
45 layers, indexer runs over the full 1,000,000-token context
reuse across 1 layer(s): 45 indexer passes (1.00x fewer)
reuse across 2 layer(s): 23 indexer passes (1.96x fewer)
reuse across 4 layer(s): 12 indexer passes (3.75x fewer)
Sharing across 4 layers removes ~75% of indexer work on its own.
IndexPool: four key vectors pooled into one
==============================================================================
The indexer scores every compressed entry. Pooling its KEYS 4:1
quarters both the entries it scores and what must be stored:
entries at 1M relative
----------------------------------------------------
no pooling 1,000,000 1.00
IndexPool 4:1 250,000 0.25
----------------------------------------------------
Combined, against the stated figures
==============================================================================
If the two applied independently and to everything, the ceiling
would be 4x and 4x respectively. The documentation reports:
attention computation 3.01x
KV cache size 4.44x
3.01x sits below the 4x indexer ceiling, which is what you would
expect: IndexShare only touches the indexer, and the core attention
over the selected top-k is unchanged.
4.44x EXCEEDS the 4x pooling ratio, so the cache saving cannot come
from IndexPool alone -- the hybrid linear-attention layers store a
fixed-size state rather than a growing cache, and that is the most
likely source of the remainder.
Stated as an inference, not a result: the numbers are consistent
with the described mechanisms, and nothing published lets anyone
check them properly.
Active-parameter trend across the family
==============================================================================
model total active active %
------------------------------------------------
GLM-4.5 355B 32B 9.0%
GLM-5 744B 40B 5.4%
GLM-5.3-Flash 320B 18B 5.6%
------------------------------------------------
Sparsity is rising across the family: GLM-4.5 activated 9.0%,
GLM-5 5.4%, Flash 5.6%. Compare [[deepseek-v4]]:
V4-Flash activates 4.6% and V4-Pro 3.1%. The whole field is
moving the same direction -- more total capacity, less of it
touched per token.
6. Three architectures, side by side
Having now worked through [[kimi-k3]], [[deepseek-v4]] and GLM-5, the comparison is the most useful thing on this page.
| Kimi K3 | DeepSeek-V4 | GLM-5 | |
|---|---|---|---|
| long-context attention | Kimi Delta Attention (linear, gated decay) | CSA + HCA (compress, then select) | DSA (select, no compression) |
| how it got there | trained in | trained in | retrofitted by continued pre-training |
| residual stream | attention residuals across blocks | mHC | standard |
| optimiser fix for logits | QK-Clip | QK RMSNorm | Muon Split |
| MoE | Stable LatentMoE | DeepSeekMoE, hash routing early | 256 experts |
Three labs, three answers to one problem, and the disagreements are informative:
On compression. DeepSeek compresses KV entries and then selects among them. GLM-5's ablation argues compression is exactly what costs you retrieval accuracy, and that selection alone — lossless by construction — is preferable. Both ship.
On stability. All three hit exploding attention logits. Kimi clips them, DeepSeek normalises them away, GLM-5 removes the cause by changing what the optimiser treats as one matrix. The third is the only one that costs nothing at inference.
On getting there. GLM-5's cheapest idea is not an architecture at all. DSA was retrofitted onto a dense model for about 22.8B tokens where DeepSeek-V3.2 spent 943.7B — over 40× cheaper, for the same mechanism at reportedly the same quality.
Verified in this notebook
- The efficient-attention ablation reproduced and ranked; interleaved SWA collapses to 6.51 on RULER@128K while the searched pattern, with an identical layer budget, holds 53.95
- Continued-pretraining cost computed from the report's own figures: ~22B tokens against DeepSeek-V3.2's 943.7B
- Muon Split contrasted with whole-matrix orthogonalisation: the weakest head's update block carries singular values far from 1 under standard Muon and exactly 1 under Split
- The MLA-256 head-shape trade computed under both readings of "decrease by 1/3", showing which one holds parameters constant
- Accept lengths 2.55 and 2.76 inverted to per-token acceptance probabilities and substituted back to confirm
- The GLM-5.3-Flash efficiency claims checked for consistency against the mechanisms described, with the part that does not follow from them identified
A note on sources
Sections 1–4 and 6 come from the GLM-5 technical report, read directly. Section 5 comes from Z.ai's GLM-5.3-Flash documentation, which is a product page — no architecture paper exists for that model, so nothing there is independently checkable, and the notebook says so rather than presenting it at the same confidence as the rest.
References
- GLM-5: from Vibe Coding to Agentic Engineering
- GLM-5.3-Flash documentation (product page - no paper exists for this model)
- DeepSeek-V3.2 / DeepSeek Sparse Attention
- Gated Delta Networks: Improving Mamba2 with Delta Rule
- Jet-Nemotron / PostNAS (search-based attention layer selection)
- RULER: What is the Real Context Size of Your Long-Context Language Models?