🧠 BERT on EEG
Bidirectional masked reconstruction - the same encoder, no causal mask, a signal instead of text
Transformers & Attention
Intermediate
Contents
The autoregressive Transformer walkthrough built a model that predicts the next token, and had to hide the future behind a causal mask to do it.
BERT throws that mask away.
Its training task is not "what comes next" but "what is missing": some inputs are blanked out, and the model reconstructs them using context from both sides. That single change — bidirectional instead of causal — is what makes BERT an encoder of meaning rather than a generator of sequences.
Here the inputs are not words but EEG. A raw multi-channel recording is cut into short time patches, each patch is projected into a vector, some patches are masked, and the encoder reconstructs them. This is exactly how EEG foundation models are pre-trained, and it maps onto BERT with one substitution:
| BERT on text | BERT on EEG |
|---|---|
| token → embedding lookup | patch → linear projection |
| predict token ID, cross-entropy | predict the waveform, MSE |
| vocabulary softmax | reconstruction head |
Everything below is computed in NumPy with every matrix printed, and each hand computation is checked against the array it came from.
Display helpers ready: banner, step, check, expand, show_matrix, show_matmul, show_elementwise.
1. The Signal
Two channels of synthetic EEG, one second at 64 Hz. Both carry a background alpha rhythm (~10 Hz); a spike-wave burst is injected in the middle, which is the interesting event a model would need to represent.
EEG array shape: (2, 64) (2 channels x 64 samples)
sampling rate 64 Hz, duration 1.0s, spike burst at samples 24-40
sample time(s) Fp1 Fp2
----------------------------------------
0 0.0000 0.0248 0.4272
1 0.0156 0.6583 0.6641
2 0.0312 0.7715 0.2725
3 0.0469 0.2322 -0.2394
4 0.0625 -0.5774 -0.5797
5 0.0781 -0.7963 -0.4069
...
28 0.4375 1.7602 0.9957
29 0.4531 1.2208 0.6281
30 0.4688 0.4369 0.3733
31 0.4844 0.1165 0.3113
32 0.5000 -0.0007 0.4013
33 0.5156 -0.0179 0.1052
----------------------------------------
mean +0.0234, std 0.5996
The dotted lines cut the recording into 8 patches of 8 samples each. Those patches are this model's 'tokens'.
2. Patching — Turning a Signal into Tokens
A Transformer needs a sequence of vectors. Feeding one time sample per position would give 64 positions carrying almost no information each, and attention cost grows with the square of that number.
So the signal is cut into non-overlapping patches, and each patch is flattened across channels:
$$\text{patch}_p \in \mathbb{R}^{C \times L} \;\longrightarrow\; \mathbb{R}^{C \cdot L}$$
with $C = 2$ channels and $L = 8$ samples, giving vectors of length 16.
8 patches x 16 values -> patches.shape = (8, 16)
patch p3 covers samples 24..31 (t = 0.375s to 0.484s)
Fp1 values: [-0.8272 -0.3474 0.638 1.6286 1.7602 1.2208 0.4369 0.1165]
Fp2 values: [-0.4854 0.1653 0.8491 1.2341 0.9957 0.6281 0.3733 0.3113]
flattened : [-0.8272 -0.3474 0.638 1.6286 1.7602 1.2208 0.4369 0.1165 -0.4854 0.1653 0.8491 1.2341 0.9957 0.6281 0.3733 0.3113]
MATCH: True
patches (8, 16) =
p0 | +0.025 +0.658 +0.771 +0.232 -0.577 -0.796 -0.227 +0.483 +0.427 +0.664 +0.272 -0.239 -0.580 -0.407 +0.200 +0.653
p1 | +0.777 +0.472 -0.329 -0.808 -0.554 +0.060 +0.653 +0.637 +0.457 +0.012 -0.664 -0.484 -0.047 +0.454 +0.576 +0.067
p2 | -0.051 -0.649 -0.785 -0.227 +0.639 +0.773 +0.310 -0.516 -0.398 -0.578 -0.202 +0.264 +0.557 +0.350 -0.136 -0.560
p3 | -0.827 -0.347 +0.638 +1.629 +1.760 +1.221 +0.437 +0.117 -0.485 +0.165 +0.849 +1.234 +0.996 +0.628 +0.373 +0.311
p4 | -0.001 -0.018 -0.220 -0.930 -1.321 -1.278 -0.474 +0.454 +0.401 +0.105 -0.524 -1.121 -1.281 -0.712 +0.083 +0.536
p5 | +0.837 +0.453 -0.312 -0.800 -0.640 +0.120 +0.716 +0.718 +0.451 -0.046 -0.438 -0.517 -0.038 +0.465 +0.476 +0.165
p6 | +0.017 -0.753 -0.723 -0.175 +0.532 +0.815 +0.358 -0.398 -0.384 -0.473 -0.286 +0.305 +0.596 +0.316 -0.124 -0.539
p7 | -0.842 -0.460 +0.323 +0.833 +0.542 -0.165 -0.794 -0.725 -0.419 +0.021 +0.603 +0.455 +0.081 -0.359 -0.621 -0.195
row energy (how active each patch is):
p0: 2.0146 ########
p1: 2.0391 ########
p2: 1.9563 #######
p3: 3.5758 ############## <- spike burst
p4: 2.9780 ########### <- spike burst
p5: 2.0568 ########
p6: 1.9134 #######
p7: 2.1221 ########
3. Patch Embedding
Text BERT looks a token ID up in an embedding table. There is no table here — the input is continuous — so the patch is projected instead:
$$E_p = \text{patch}_p W_e + b_e, \qquad W_e \in \mathbb{R}^{16 \times 8}$$
This is the only structural difference at the input, and it is why the same architecture transfers to signals, images (ViT), and audio.
==============================================================================
patches (8, 16) @ W_e (16, 8) = E_patch (8, 8)
==============================================================================
shape rule: (8 x 16) @ (16 x 8) -> (8 x 8) inner dim 16 cancels
each 16-value patch becomes an 8-dim vector
==============================================================================
patches (8, 16) [first 14 of 16 columns] =
p0 | +0.025 +0.658 +0.771 +0.232 -0.577 -0.796 -0.227 +0.483 +0.427 +0.664 +0.272 -0.239 -0.580 -0.407
p1 | +0.777 +0.472 -0.329 -0.808 -0.554 +0.060 +0.653 +0.637 +0.457 +0.012 -0.664 -0.484 -0.047 +0.454
p2 | -0.051 -0.649 -0.785 -0.227 +0.639 +0.773 +0.310 -0.516 -0.398 -0.578 -0.202 +0.264 +0.557 +0.350
p3 | -0.827 -0.347 +0.638 +1.629 +1.760 +1.221 +0.437 +0.117 -0.485 +0.165 +0.849 +1.234 +0.996 +0.628
p4 | -0.001 -0.018 -0.220 -0.930 -1.321 -1.278 -0.474 +0.454 +0.401 +0.105 -0.524 -1.121 -1.281 -0.712
p5 | +0.837 +0.453 -0.312 -0.800 -0.640 +0.120 +0.716 +0.718 +0.451 -0.046 -0.438 -0.517 -0.038 +0.465
p6 | +0.017 -0.753 -0.723 -0.175 +0.532 +0.815 +0.358 -0.398 -0.384 -0.473 -0.286 +0.305 +0.596 +0.316
p7 | -0.842 -0.460 +0.323 +0.833 +0.542 -0.165 -0.794 -0.725 -0.419 +0.021 +0.603 +0.455 +0.081 -0.359
W_e (16, 8) =
0 | -0.459 +0.091 +0.178 -0.483 +0.012 -0.274 +0.145 -0.326
1 | +0.191 -0.113 +0.437 -0.362 -0.159 -0.387 +0.425 +0.377
2 | -0.242 +0.160 +0.317 +0.055 +0.030 -0.258 -0.407 +0.397
3 | +0.400 +0.133 -0.161 -0.151 +0.226 +0.397 +0.387 +0.280
4 | +0.142 -0.416 -0.338 +0.399 +0.106 -0.491 -0.399 +0.164
5 | -0.495 -0.339 +0.049 +0.192 +0.152 -0.276 +0.212 -0.263
6 | -0.175 +0.246 +0.150 +0.349 +0.158 +0.068 -0.406 -0.132
7 | -0.235 -0.256 +0.473 -0.107 +0.392 +0.131 +0.295 +0.003
8 | +0.077 -0.007 -0.305 +0.222 -0.219 -0.476 +0.145 -0.323
9 | +0.440 +0.454 +0.415 -0.130 -0.485 +0.428 -0.072 +0.467
10 | +0.464 +0.353 -0.206 -0.115 +0.351 -0.183 -0.331 +0.057
11 | +0.436 +0.196 +0.070 -0.403 +0.115 +0.490 -0.360 +0.018
12 | +0.377 +0.241 +0.197 +0.202 -0.141 -0.206 +0.309 +0.310
13 | +0.367 +0.413 +0.011 +0.002 +0.298 +0.150 +0.202 +0.296
14 | +0.390 -0.162 -0.124 -0.406 +0.078 -0.464 -0.034 +0.043
15 | -0.213 +0.091 -0.469 -0.463 +0.323 -0.140 -0.373 +0.022
E_patch (8, 8) =
p0 | +0.176 +0.479 +0.472 -1.184 -0.219 -0.029 -0.147 +0.664
p1 | -1.002 -0.174 +0.819 -0.394 -0.141 -0.775 +0.782 -0.666
p2 | -0.085 -0.436 -0.546 +1.148 +0.222 -0.010 -0.008 -0.635
p3 | +2.006 +0.302 -0.635 +0.574 +1.563 +0.110 -0.984 +1.579
p4 | -1.383 -0.335 +0.029 -0.806 -0.750 +0.168 +0.188 -0.894
p5 | -1.094 -0.073 +0.803 -0.436 +0.038 -0.791 +0.736 -0.730
p6 | -0.174 -0.367 -0.412 +1.094 +0.215 +0.129 +0.024 -0.648
p7 | +1.175 +0.227 -0.810 +0.460 -0.000 +0.853 -0.647 +0.773
worked element E_patch[3,0] = row 3 of patches . column 0 of W_e
row 3 of patches = [-0.8272 -0.3474 0.638 1.6286 1.7602 1.2208 0.4369 0.1165 -0.4854 0.1653 0.8491 1.2341 0.9957 0.6281 0.3733 0.3113]
col 0 of W_e = [-0.4592 0.1909 -0.2421 0.4004 0.142 -0.4949 -0.1746 -0.2348 0.0769 0.4405 0.4636 0.4362 0.3774 0.3671 0.39 -0.2135]
(-0.8272)*(-0.4592) + (-0.3474)*(+0.1909) + (+0.6380)*(-0.2421) + (+1.6286)*(+0.4004) +
(+1.7602)*(+0.1420) + (+1.2208)*(-0.4949) + (+0.4369)*(-0.1746) + (+0.1165)*(-0.2348) +
(-0.4854)*(+0.0769) + (+0.1653)*(+0.4405) + (+0.8491)*(+0.4636) + (+1.2341)*(+0.4362) +
(+0.9957)*(+0.3774) + (+0.6281)*(+0.3671) + (+0.3733)*(+0.3900) + (+0.3113)*(-0.2135)
= +2.006270 | from array: +2.006270 | MATCH: True
The [CLS] token and positional encoding
Two additions, both borrowed unchanged from text BERT:
[CLS] is a learned vector prepended to the sequence. It belongs to no patch, so it
is free to accumulate a summary of the whole recording through attention — which is
what a downstream classifier reads.
Positional encoding restores order. Attention is permutation-invariant: without it, a spike at 0.4s and the same spike at 0.9s would be indistinguishable.
$$Z = \big[\,\texttt{[CLS]};\, E_0;\, \dots;\, E_{7}\,\big] + PE$$
sequence length = 1 ([CLS]) + 8 patches = 9
[CLS] vector = [-0.2529 0.2403 -0.2582 0.0765 0.0161 -0.2304 0.758 0.2241]
==============================================================================
Z0 = embeddings + PE all three are (9, 8) - element-wise
==============================================================================
[CLS] |
embeddings: -0.253 +0.240 -0.258 +0.077 +0.016 -0.230 +0.758 +0.224
PE: +0.000 +1.000 +0.000 +1.000 +0.000 +1.000 +0.000 +1.000
Z0: -0.253 +1.240 -0.258 +1.077 +0.016 +0.770 +0.758 +1.224
p0 |
embeddings: +0.176 +0.479 +0.472 -1.184 -0.219 -0.029 -0.147 +0.664
PE: +0.841 +0.540 +0.100 +0.995 +0.010 +1.000 +0.001 +1.000
Z0: +1.018 +1.020 +0.572 -0.189 -0.209 +0.971 -0.146 +1.664
p1 |
embeddings: -1.002 -0.174 +0.819 -0.394 -0.141 -0.775 +0.782 -0.666
PE: +0.909 -0.416 +0.199 +0.980 +0.020 +1.000 +0.002 +1.000
Z0: -0.092 -0.590 +1.018 +0.586 -0.121 +0.225 +0.784 +0.334
p2 |
embeddings: -0.085 -0.436 -0.546 +1.148 +0.222 -0.010 -0.008 -0.635
PE: +0.141 -0.990 +0.296 +0.955 +0.030 +1.000 +0.003 +1.000
Z0: +0.056 -1.426 -0.251 +2.104 +0.252 +0.990 -0.005 +0.365
p3 |
embeddings: +2.006 +0.302 -0.635 +0.574 +1.563 +0.110 -0.984 +1.579
PE: -0.757 -0.654 +0.389 +0.921 +0.040 +0.999 +0.004 +1.000
Z0: +1.249 -0.351 -0.245 +1.495 +1.603 +1.109 -0.980 +2.579
p4 |
embeddings: -1.383 -0.335 +0.029 -0.806 -0.750 +0.168 +0.188 -0.894
PE: -0.959 +0.284 +0.479 +0.878 +0.050 +0.999 +0.005 +1.000
Z0: -2.342 -0.051 +0.508 +0.071 -0.700 +1.166 +0.193 +0.106
p5 |
embeddings: -1.094 -0.073 +0.803 -0.436 +0.038 -0.791 +0.736 -0.730
PE: -0.279 +0.960 +0.565 +0.825 +0.060 +0.998 +0.006 +1.000
Z0: -1.374 +0.887 +1.368 +0.389 +0.098 +0.207 +0.742 +0.270
p6 |
embeddings: -0.174 -0.367 -0.412 +1.094 +0.215 +0.129 +0.024 -0.648
PE: +0.657 +0.754 +0.644 +0.765 +0.070 +0.998 +0.007 +1.000
Z0: +0.483 +0.387 +0.232 +1.859 +0.285 +1.126 +0.031 +0.352
p7 |
embeddings: +1.175 +0.227 -0.810 +0.460 -0.000 +0.853 -0.647 +0.773
PE: +0.989 -0.146 +0.717 +0.697 +0.080 +0.997 +0.008 +1.000
Z0: +2.164 +0.082 -0.093 +1.156 +0.080 +1.849 -0.639 +1.773
check [0,0]: -0.252898 + +0.000000 = -0.252898 MATCH: True
4. Masking
BERT's training signal comes from hiding part of the input. A fraction of positions is
selected, and the chosen ones are replaced by a single learned [MASK] vector.
Text BERT applies the well-known 80/10/10 rule to each selected position:
- 80% replaced by
[MASK], - 10% replaced by a random token,
- 10% left unchanged.
The reason is a train/inference mismatch: [MASK] never appears when the model is
actually used, so if every selected position were masked the model would only ever learn
to handle that symbol. The 10% "unchanged" slice forces it to keep a good representation
of every position, because it cannot tell which ones it will be scored on.
masking 25% of 8 patches -> 2 positions
[MASK] vector = [-0.716 0.0659 -0.234 0.3014 -0.2802 -0.0406 0.1785 0.3061]
pos name masked input row (after masking)
--------------------------------------------------------------
0 [CLS] - [-0.253 1.24 -0.258 1.077 0.016 0.77 0.758 1.224]
1 p0 - [ 1.018 1.02 0.572 -0.189 -0.209 0.971 -0.146 1.664]
2 p1 - [-0.092 -0.59 1.018 0.586 -0.121 0.225 0.784 0.334]
3 p2 - [ 0.056 -1.426 -0.251 2.104 0.252 0.99 -0.005 0.365]
4 p3 YES [-1.473 -0.588 0.155 1.222 -0.24 0.959 0.183 1.306]
5 p4 - [-2.342 -0.051 0.508 0.071 -0.7 1.166 0.193 0.106]
6 p5 - [-1.374 0.887 1.368 0.389 0.098 0.207 0.742 0.27 ]
7 p6 - [0.483 0.387 0.232 1.859 0.285 1.126 0.031 0.352]
8 p7 YES [ 0.273 -0.08 0.483 0.998 -0.2 0.956 0.187 1.306]
--------------------------------------------------------------
masked positions: [4, 8] = ['p3', 'p7']
The true content of those patches is kept aside as the reconstruction
target - the encoder never sees it again until the loss is computed.
targets.shape = (2, 16) (2 patches x 16 values)
5. Bidirectional Self-Attention
This is the difference that defines BERT.
The autoregressive decoder added a causal mask $M$ before the softmax so position $t$ could not see beyond itself. BERT adds nothing:
$$A = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) \qquad\text{(no } +M \text{ term)}$$
Every position attends to every other position, in both directions. A masked patch is reconstructed from the signal before and after it — which is why BERT is useless for generation and excellent for representation.
STEP 1 : project Z to Q, K, V
==============================================================================
==============================================================================
Z (9, 8) @ Wq (8, 8) = Q_full (9, 8)
==============================================================================
shape rule: (9 x 8) @ (8 x 8) -> (9 x 8) inner dim 8 cancels
K uses Wk, V uses Wv - identical arithmetic
==============================================================================
Z (9, 8) =
[CLS] | -0.253 +1.240 -0.258 +1.077 +0.016 +0.770 +0.758 +1.224
p0 | +1.018 +1.020 +0.572 -0.189 -0.209 +0.971 -0.146 +1.664
p1 | -0.092 -0.590 +1.018 +0.586 -0.121 +0.225 +0.784 +0.334
p2 | +0.056 -1.426 -0.251 +2.104 +0.252 +0.990 -0.005 +0.365
p3 | -1.473 -0.588 +0.155 +1.222 -0.240 +0.959 +0.183 +1.306
p4 | -2.342 -0.051 +0.508 +0.071 -0.700 +1.166 +0.193 +0.106
p5 | -1.374 +0.887 +1.368 +0.389 +0.098 +0.207 +0.742 +0.270
p6 | +0.483 +0.387 +0.232 +1.859 +0.285 +1.126 +0.031 +0.352
p7 | +0.273 -0.080 +0.483 +0.998 -0.200 +0.956 +0.187 +1.306
Wq (8, 8) =
0 | -0.319 -0.497 -0.388 +0.532 +0.169 +0.020 +0.192 -0.079
1 | +0.282 -0.554 +0.081 -0.418 -0.465 -0.194 -0.500 -0.497
2 | -0.231 +0.587 -0.398 -0.591 +0.323 +0.376 -0.188 -0.043
3 | +0.183 -0.554 +0.550 +0.474 -0.293 -0.594 +0.531 +0.001
4 | +0.048 +0.225 +0.142 +0.544 +0.544 +0.450 +0.167 +0.369
5 | +0.217 +0.090 -0.455 +0.381 +0.393 +0.154 +0.392 +0.186
6 | -0.359 -0.277 -0.350 -0.150 -0.565 +0.145 -0.200 +0.191
7 | -0.140 +0.222 -0.195 -0.293 -0.005 +0.236 -0.186 +0.535
Q_full (9, 8) =
[CLS] | +0.411 -1.174 +0.042 -0.161 -1.141 -0.457 -0.123 +0.364
p0 | -0.185 -0.180 -1.390 -0.522 +0.280 +0.579 -0.456 +0.353
p1 | -0.550 +0.497 -0.553 -0.323 -0.006 +0.320 +0.246 +0.583
p2 | +0.202 -0.321 +0.636 +2.178 +0.502 -0.714 +2.251 +1.189
p3 | +0.440 +0.745 +0.346 -0.227 -0.147 -0.209 +0.687 +1.226
p4 | +0.763 +1.369 +0.024 -1.488 -0.262 +0.029 -0.200 +0.240
p5 | +0.188 +0.675 -0.117 -1.785 -0.604 +0.330 -0.860 -0.031
p6 | +0.440 -1.112 +0.222 +1.315 +0.011 -0.692 +1.261 +0.270
p7 | -0.091 -0.081 -0.539 +0.210 +0.102 +0.003 +0.593 +0.836
worked element Q_full[0,0] = row 0 of Z . column 0 of Wq
row 0 of Z = [-0.2529 1.2403 -0.2582 1.0765 0.0161 0.7696 0.758 1.2241]
col 0 of Wq = [-0.3187 0.2817 -0.231 0.1834 0.0482 0.217 -0.3592 -0.1404]
(-0.2529)*(-0.3187) + (+1.2403)*(+0.2817) + (-0.2582)*(-0.2310) + (+1.0765)*(+0.1834) +
(+0.0161)*(+0.0482) + (+0.7696)*(+0.2170) + (+0.7580)*(-0.3592) + (+1.2241)*(-0.1404)
= +0.410802 | from array: +0.410802 | MATCH: True
K_full (9, 8), V_full (9, 8)
STEP 2 : split into 2 heads of 4 -> (2, 9, 4)
STEP 3 : scores = Q K^T / sqrt(d_k) (head 0)
==============================================================================
==============================================================================
Q_head0 (9, 4) @ K_head0^T (4, 9) = raw_scores (9, 9)
==============================================================================
shape rule: (9 x 4) @ (4 x 9) -> (9 x 9) inner dim 4 cancels
(9,4) @ (4,9) -> (9,9)
==============================================================================
Q_head0 (9, 4) =
[CLS] | +0.411 -1.174 +0.042 -0.161
p0 | -0.185 -0.180 -1.390 -0.522
p1 | -0.550 +0.497 -0.553 -0.323
p2 | +0.202 -0.321 +0.636 +2.178
p3 | +0.440 +0.745 +0.346 -0.227
p4 | +0.763 +1.369 +0.024 -1.488
p5 | +0.188 +0.675 -0.117 -1.785
p6 | +0.440 -1.112 +0.222 +1.315
p7 | -0.091 -0.081 -0.539 +0.210
K_head0^T (4, 9) =
0 | +1.036 -0.694 +0.844 +2.334 +2.509 +2.331 +0.533 +1.051 +1.041
1 | +0.552 -0.626 +0.651 +1.874 +1.032 +0.878 +0.211 +1.207 +0.562
2 | -0.206 +0.305 +0.055 -0.981 -1.444 -1.927 -0.483 -0.552 -0.334
3 | +1.840 +1.519 +0.238 +0.902 +1.260 +0.492 +0.609 +1.622 +1.449
raw_scores (9, 9) =
[CLS] | -0.527 +0.217 -0.453 -1.427 -0.443 -0.232 -0.146 -1.270 -0.479
p0 | -0.964 -0.975 -0.474 +0.125 +0.701 +1.834 +0.217 -0.490 -0.584
p1 | -0.776 -0.588 -0.248 -0.101 -0.476 +0.061 -0.118 -0.196 -0.577
p2 | +3.908 +3.563 +0.515 +1.209 +2.001 +0.035 +1.059 +3.006 +2.973
p3 | +0.378 -1.010 +0.821 +1.878 +1.087 +0.900 +0.086 +0.803 +0.432
p4 | -1.196 -3.638 +1.182 +2.980 +1.417 +2.201 -0.222 +0.028 -0.600
p5 | -2.692 -3.300 +0.167 +0.210 -0.910 +0.379 -0.788 -1.817 -1.972
p6 | +2.216 +2.456 -0.027 -0.089 +1.292 +0.268 +0.694 +1.130 +1.664
p7 | +0.360 +0.269 -0.109 +0.356 +0.733 +0.861 +0.323 +0.446 +0.345
worked element raw_scores[0,0] = row 0 of Q_head0 . column 0 of K_head0^T
row 0 of Q_head0 = [ 0.4108 -1.1735 0.0416 -0.1612]
col 0 of K_head0^T = [ 1.0363 0.5522 -0.2061 1.8397]
(+0.4108)*(+1.0363) + (-1.1735)*(+0.5522) + (+0.0416)*(-0.2061) + (-0.1612)*(+1.8397)
= -0.527471 | from array: -0.527471 | MATCH: True
then divide every entry by sqrt(d_k) = 2:
scaled_scores (9, 9) =
[CLS] | -0.264 +0.109 -0.227 -0.714 -0.222 -0.116 -0.073 -0.635 -0.239
p0 | -0.482 -0.487 -0.237 +0.063 +0.350 +0.917 +0.109 -0.245 -0.292
p1 | -0.388 -0.294 -0.124 -0.050 -0.238 +0.031 -0.059 -0.098 -0.288
p2 | +1.954 +1.781 +0.258 +0.605 +1.000 +0.017 +0.530 +1.503 +1.486
p3 | +0.189 -0.505 +0.411 +0.939 +0.543 +0.450 +0.043 +0.401 +0.216
p4 | -0.598 -1.819 +0.591 +1.490 +0.709 +1.100 -0.111 +0.014 -0.300
p5 | -1.346 -1.650 +0.083 +0.105 -0.455 +0.190 -0.394 -0.909 -0.986
p6 | +1.108 +1.228 -0.013 -0.045 +0.646 +0.134 +0.347 +0.565 +0.832
p7 | +0.180 +0.134 -0.054 +0.178 +0.366 +0.430 +0.162 +0.223 +0.173
STEP 4 : softmax - and note what is NOT here
==============================================================================
A = softmax(scores) (9, 9) =
[CLS] | 0.1081 0.1568 0.1121 0.0689 0.1127 0.1253 0.1307 0.0746 0.1107
p0 | 0.0640 0.0636 0.0818 0.1103 0.1471 0.2592 0.1155 0.0811 0.0774
p1 | 0.0884 0.0971 0.1151 0.1239 0.1027 0.1343 0.1228 0.1181 0.0976
p2 | 0.2316 0.1949 0.0425 0.0601 0.0892 0.0334 0.0557 0.1475 0.1451
p3 | 0.0933 0.0466 0.1165 0.1975 0.1330 0.1212 0.0806 0.1154 0.0959
p4 | 0.0376 0.0111 0.1233 0.3031 0.1387 0.2053 0.0611 0.0693 0.0506
p5 | 0.0438 0.0323 0.1829 0.1869 0.1067 0.2034 0.1135 0.0678 0.0628
p6 | 0.1791 0.2019 0.0584 0.0566 0.1129 0.0676 0.0837 0.1040 0.1359
p7 | 0.1081 0.1033 0.0855 0.1079 0.1302 0.1388 0.1061 0.1128 0.1073
row sums (must all be 1): [1. 1. 1. 1. 1. 1. 1. 1. 1.]
entries ABOVE the diagonal - the 'future' - are all non-zero: True
smallest such weight = 0.033393
In the autoregressive decoder every one of those was exactly 0.
That is the whole difference between BERT and GPT, visible as a matrix.
Bidirectional vs causal, side by side
The same scores, softmaxed twice: once as BERT does it, once with the causal mask the decoder-style model would apply.
query BERT: non-zero weights causal: non-zero weights
--------------------------------------------------------------
[CLS] 9 1
p0 9 2
p1 9 3
p2 9 4
p3 9 5
p4 9 6
p5 9 7
p6 9 8
p7 9 9
--------------------------------------------------------------
Every BERT row uses all 9 positions. The causal rows grow 1, 2, 3, ...
A masked patch in the middle of the recording therefore gets context from
BOTH sides under BERT, and only from the left under a causal model.
One attention weight, by hand
$$A_{ij} = \frac{\exp(s_{ij} - m)}{\sum_{j'} \exp(s_{ij'} - m)}, \qquad s_{ij} = \frac{q_i \cdot k_j}{\sqrt{d_k}}, \qquad m = \max_{j} s_{ij}$$
query = position 4 ('p3', MASKED), key = position 5 ('p4')
q = [ 0.4397 0.7448 0.3462 -0.2267]
k = [ 2.331 0.878 -1.9271 0.492 ]
q . k = (+0.4397)*(+2.3310) + (+0.7448)*(+0.8780) + (+0.3462)*(-1.9271) + (-0.2267)*(+0.4920)
= +0.900163
/ sqrt(d_k) = +0.900163 / 2 = +0.450082
from the scores matrix: +0.450082 MATCH: True
softmax over row 4, m = max = 0.938971
j name s_ij s-m exp /Z array
----------------------------------------------------------------------
0 [CLS] 0.189236 -0.749735 0.472492 0.093338 0.093338
1 p0 -0.504971 -1.443941 0.235996 0.046619 0.046619
2 p1 0.410514 -0.528456 0.589514 0.116455 0.116455
3 p2 0.938971 0.000000 1.000000 0.197544 0.197544
4 p3 0.543256 -0.395714 0.673199 0.132986 0.132986
5 p4 0.450082 -0.488889 0.613307 0.121155 0.121155
6 p5 0.043121 -0.895849 0.408261 0.080649 0.080649
7 p6 0.401259 -0.537711 0.584083 0.115382 0.115382
8 p7 0.216020 -0.722951 0.485318 0.095872 0.095872
----------------------------------------------------------------------
Z = 5.062170, sum of weights = 1.0000000000
MATCH with the array row: True
6. The Encoder Block
Identical in form to the encoder half of the autoregressive Transformer: attention, residual + LayerNorm, feed-forward, residual + LayerNorm. Nothing here is BERT-specific — the only BERT-specific choice was the absent mask.
STEPS 5-7 : weights @ V, concat heads, project with W_O
==============================================================================
==============================================================================
A_head0 (9, 9) @ V_head0 (9, 4) = head0_out (9, 4)
==============================================================================
shape rule: (9 x 9) @ (9 x 4) -> (9 x 4) inner dim 9 cancels
(9,9) @ (9,4) -> (9,4)
==============================================================================
A_head0 (9, 9) =
[CLS] | +0.108 +0.157 +0.112 +0.069 +0.113 +0.125 +0.131 +0.075 +0.111
p0 | +0.064 +0.064 +0.082 +0.110 +0.147 +0.259 +0.115 +0.081 +0.077
p1 | +0.088 +0.097 +0.115 +0.124 +0.103 +0.134 +0.123 +0.118 +0.098
p2 | +0.232 +0.195 +0.042 +0.060 +0.089 +0.033 +0.056 +0.148 +0.145
p3 | +0.093 +0.047 +0.116 +0.198 +0.133 +0.121 +0.081 +0.115 +0.096
p4 | +0.038 +0.011 +0.123 +0.303 +0.139 +0.205 +0.061 +0.069 +0.051
p5 | +0.044 +0.032 +0.183 +0.187 +0.107 +0.203 +0.113 +0.068 +0.063
p6 | +0.179 +0.202 +0.058 +0.057 +0.113 +0.068 +0.084 +0.104 +0.136
p7 | +0.108 +0.103 +0.085 +0.108 +0.130 +0.139 +0.106 +0.113 +0.107
V_head0 (9, 4) =
0 | +0.158 -1.394 -1.264 -0.123
1 | +0.197 -0.766 -0.639 -1.657
2 | -0.134 -0.605 -0.688 -0.961
3 | +1.146 -0.971 -1.650 +0.281
4 | +1.222 -1.446 -1.388 -0.736
5 | +1.177 -1.143 -0.061 -0.986
6 | -0.564 -1.140 -0.446 -0.428
7 | +0.398 -1.170 -1.403 +0.329
8 | +0.609 -1.065 -1.276 -1.107
head0_out (9, 4) =
[CLS] | +0.420 -1.066 -0.896 -0.722
p0 | +0.637 -1.112 -0.844 -0.633
p1 | +0.480 -1.072 -0.953 -0.578
p2 | +0.402 -1.114 -1.088 -0.610
p3 | +0.599 -1.088 -1.066 -0.465
p4 | +0.774 -1.069 -1.034 -0.420
p5 | +0.574 -1.040 -0.897 -0.557
p6 | +0.420 -1.097 -1.006 -0.699
p7 | +0.522 -1.100 -0.971 -0.596
worked element head0_out[0,0] = row 0 of A_head0 . column 0 of V_head0
row 0 of A_head0 = [0.1081 0.1568 0.1121 0.0689 0.1127 0.1253 0.1307 0.0746 0.1107]
col 0 of V_head0 = [ 0.1577 0.1974 -0.134 1.1456 1.222 1.1771 -0.564 0.398 0.6086]
(+0.1081)*(+0.1577) + (+0.1568)*(+0.1974) + (+0.1121)*(-0.1340) + (+0.0689)*(+1.1456) +
(+0.1127)*(+1.2220) + (+0.1253)*(+1.1771) + (+0.1307)*(-0.5640) + (+0.0746)*(+0.3980) +
(+0.1107)*(+0.6086)
= +0.420451 | from array: +0.420451 | MATCH: True
==============================================================================
concat (9, 8) @ W_O (8, 8) = attn_out (9, 8)
==============================================================================
shape rule: (9 x 8) @ (8 x 8) -> (9 x 8) inner dim 8 cancels
W_O mixes the two heads
==============================================================================
concat (9, 8) =
[CLS] | +0.420 -1.066 -0.896 -0.722 +0.531 -0.615 +0.610 +0.172
p0 | +0.637 -1.112 -0.844 -0.633 +0.649 -0.793 +0.586 -0.061
p1 | +0.480 -1.072 -0.953 -0.578 +0.727 -0.733 +0.598 +0.057
p2 | +0.402 -1.114 -1.088 -0.610 +1.059 -0.599 +0.614 +0.396
p3 | +0.599 -1.088 -1.066 -0.465 +0.874 -0.649 +0.610 +0.267
p4 | +0.774 -1.069 -1.034 -0.420 +0.602 -0.704 +0.601 +0.058
p5 | +0.574 -1.040 -0.897 -0.557 +0.464 -0.712 +0.586 +0.010
p6 | +0.420 -1.097 -1.006 -0.699 +0.816 -0.647 +0.616 +0.202
p7 | +0.522 -1.100 -0.971 -0.596 +0.820 -0.692 +0.607 +0.161
W_O (8, 8) =
0 | -0.511 +0.461 +0.063 -0.410 -0.109 +0.340 -0.024 +0.594
1 | -0.151 +0.306 -0.131 +0.403 +0.085 -0.535 -0.567 -0.448
2 | -0.596 -0.520 +0.235 +0.042 +0.306 +0.506 +0.104 +0.277
3 | +0.315 -0.150 -0.317 -0.361 -0.304 -0.276 -0.359 +0.463
4 | +0.315 -0.555 -0.283 -0.585 -0.002 -0.029 +0.406 -0.235
5 | +0.387 +0.573 -0.504 +0.357 +0.110 -0.024 -0.097 +0.349
6 | +0.171 +0.374 +0.494 +0.144 +0.588 +0.132 +0.167 +0.067
7 | -0.501 +0.277 +0.058 -0.060 +0.503 -0.247 +0.029 +0.242
attn_out (9, 8) =
[CLS] | +0.199 +0.071 +0.656 -0.832 +0.186 +0.496 +1.142 -0.111
p0 | +0.173 -0.126 +0.691 -1.092 -0.004 +0.652 +1.192 -0.054
p1 | +0.321 -0.108 +0.592 -1.066 +0.040 +0.475 +1.172 -0.138
p2 | +0.426 -0.090 +0.438 -1.209 +0.207 +0.317 +1.330 -0.167
p3 | +0.340 +0.012 +0.474 -1.233 +0.078 +0.379 +1.186 +0.006
p4 | +0.240 +0.134 +0.564 -1.161 -0.059 +0.493 +1.046 +0.126
p5 | +0.187 +0.052 +0.656 -0.934 +0.015 +0.530 +1.038 -0.015
p6 | +0.341 -0.051 +0.566 -1.036 +0.157 +0.436 +1.261 -0.188
p7 | +0.303 -0.081 +0.564 -1.133 +0.094 +0.473 +1.230 -0.096
worked element attn_out[0,0] = row 0 of concat . column 0 of W_O
row 0 of concat = [ 0.4205 -1.0659 -0.896 -0.7223 0.5309 -0.6145 0.61 0.1724]
col 0 of W_O = [-0.511 -0.151 -0.5956 0.3149 0.3148 0.3875 0.1707 -0.5009]
(+0.4205)*(-0.5110) + (-1.0659)*(-0.1510) + (-0.8960)*(-0.5956) + (-0.7223)*(+0.3149) +
(+0.5309)*(+0.3148) + (-0.6145)*(+0.3875) + (+0.6100)*(+0.1707) + (+0.1724)*(-0.5009)
= +0.199056 | from array: +0.199056 | MATCH: True
Add & Norm 1
==============================================================================
==============================================================================
residual = Z + attn_out all three are (9, 8) - element-wise
==============================================================================
[CLS] |
Z: -0.253 +1.240 -0.258 +1.077 +0.016 +0.770 +0.758 +1.224
attn_out: +0.199 +0.071 +0.656 -0.832 +0.186 +0.496 +1.142 -0.111
residual: -0.054 +1.311 +0.397 +0.244 +0.203 +1.266 +1.900 +1.113
p0 |
Z: +1.018 +1.020 +0.572 -0.189 -0.209 +0.971 -0.146 +1.664
attn_out: +0.173 -0.126 +0.691 -1.092 -0.004 +0.652 +1.192 -0.054
residual: +1.191 +0.894 +1.263 -1.282 -0.213 +1.624 +1.046 +1.609
p1 |
Z: -0.092 -0.590 +1.018 +0.586 -0.121 +0.225 +0.784 +0.334
attn_out: +0.321 -0.108 +0.592 -1.066 +0.040 +0.475 +1.172 -0.138
residual: +0.228 -0.698 +1.610 -0.480 -0.081 +0.700 +1.956 +0.196
p2 |
Z: +0.056 -1.426 -0.251 +2.104 +0.252 +0.990 -0.005 +0.365
attn_out: +0.426 -0.090 +0.438 -1.209 +0.207 +0.317 +1.330 -0.167
residual: +0.482 -1.515 +0.187 +0.895 +0.458 +1.307 +1.325 +0.199
p3 |
Z: -1.473 -0.588 +0.155 +1.222 -0.240 +0.959 +0.183 +1.306
attn_out: +0.340 +0.012 +0.474 -1.233 +0.078 +0.379 +1.186 +0.006
residual: -1.132 -0.575 +0.630 -0.011 -0.162 +1.338 +1.369 +1.312
p4 |
Z: -2.342 -0.051 +0.508 +0.071 -0.700 +1.166 +0.193 +0.106
attn_out: +0.240 +0.134 +0.564 -1.161 -0.059 +0.493 +1.046 +0.126
residual: -2.102 +0.083 +1.072 -1.090 -0.759 +1.659 +1.239 +0.232
p5 |
Z: -1.374 +0.887 +1.368 +0.389 +0.098 +0.207 +0.742 +0.270
attn_out: +0.187 +0.052 +0.656 -0.934 +0.015 +0.530 +1.038 -0.015
residual: -1.187 +0.939 +2.024 -0.545 +0.113 +0.737 +1.781 +0.255
p6 |
Z: +0.483 +0.387 +0.232 +1.859 +0.285 +1.126 +0.031 +0.352
attn_out: +0.341 -0.051 +0.566 -1.036 +0.157 +0.436 +1.261 -0.188
residual: +0.824 +0.335 +0.799 +0.823 +0.442 +1.562 +1.291 +0.163
p7 |
Z: +0.273 -0.080 +0.483 +0.998 -0.200 +0.956 +0.187 +1.306
attn_out: +0.303 -0.081 +0.564 -1.133 +0.094 +0.473 +1.230 -0.096
residual: +0.576 -0.161 +1.048 -0.135 -0.106 +1.429 +1.417 +1.210
check [0,0]: -0.252898 + +0.199056 = -0.053842 MATCH: True
Z1 = LayerNorm(residual) (9, 8) =
[CLS] | -1.317 +0.795 -0.619 -0.856 -0.920 +0.724 +1.706 +0.487
p0 | +0.450 +0.135 +0.526 -2.171 -1.038 +0.908 +0.296 +0.893
p1 | -0.227 -1.275 +1.336 -1.028 -0.577 +0.307 +1.728 -0.263
p2 | +0.077 -2.292 -0.273 +0.567 +0.049 +1.055 +1.077 -0.259
p3 | -1.645 -1.025 +0.316 -0.397 -0.565 +1.103 +1.138 +1.075
p4 | -1.775 +0.034 +0.853 -0.937 -0.663 +1.339 +0.992 +0.158
p5 | -1.665 +0.415 +1.477 -1.036 -0.393 +0.218 +1.238 -0.254
p6 | +0.099 -1.003 +0.042 +0.096 -0.763 +1.766 +1.154 -1.392
p7 | -0.126 -1.238 +0.585 -1.199 -1.155 +1.160 +1.142 +0.830
per-row mean ~ 0: [ 0. -0. -0. 0. -0. -0. 0. -0. 0.]
per-row std ~ 1: [1. 1. 1. 1. 1. 1. 1. 1. 1.]
Feed-forward network: 8 -> 16 -> 8
==============================================================================
==============================================================================
Z1 (9, 8) @ W1 (8, 16) = pre-ReLU (9, 16)
==============================================================================
shape rule: (9 x 8) @ (8 x 16) -> (9 x 16) inner dim 8 cancels
==============================================================================
Z1 (9, 8) =
[CLS] | -1.317 +0.795 -0.619 -0.856 -0.920 +0.724 +1.706 +0.487
p0 | +0.450 +0.135 +0.526 -2.171 -1.038 +0.908 +0.296 +0.893
p1 | -0.227 -1.275 +1.336 -1.028 -0.577 +0.307 +1.728 -0.263
p2 | +0.077 -2.292 -0.273 +0.567 +0.049 +1.055 +1.077 -0.259
p3 | -1.645 -1.025 +0.316 -0.397 -0.565 +1.103 +1.138 +1.075
p4 | -1.775 +0.034 +0.853 -0.937 -0.663 +1.339 +0.992 +0.158
p5 | -1.665 +0.415 +1.477 -1.036 -0.393 +0.218 +1.238 -0.254
p6 | +0.099 -1.003 +0.042 +0.096 -0.763 +1.766 +1.154 -1.392
p7 | -0.126 -1.238 +0.585 -1.199 -1.155 +1.160 +1.142 +0.830
W1 (8, 16) [first 14 of 16 columns] =
0 | +0.296 -0.041 +0.342 +0.269 -0.434 -0.454 +0.121 -0.153 -0.291 +0.080 -0.158 +0.037 -0.040 +0.085
1 | -0.320 +0.197 -0.088 +0.374 +0.015 +0.473 +0.102 -0.276 +0.322 -0.155 -0.152 -0.468 +0.049 +0.034
2 | -0.371 -0.170 -0.178 -0.408 -0.019 +0.188 +0.012 -0.343 -0.123 -0.497 +0.368 -0.415 +0.097 +0.486
3 | -0.264 +0.260 +0.031 +0.221 -0.438 -0.352 -0.367 +0.187 +0.344 +0.250 -0.470 +0.367 -0.146 -0.103
4 | -0.318 +0.064 +0.341 -0.411 +0.035 -0.267 -0.157 -0.026 -0.145 +0.149 -0.020 +0.084 +0.237 +0.058
5 | -0.121 -0.163 +0.400 +0.108 -0.256 -0.002 -0.170 +0.434 -0.492 -0.275 -0.135 -0.012 +0.351 -0.412
6 | +0.342 -0.448 -0.482 +0.197 +0.497 +0.397 +0.076 +0.417 -0.495 +0.475 -0.009 +0.223 +0.321 +0.218
7 | +0.339 -0.295 +0.468 +0.211 -0.300 +0.236 +0.030 +0.207 +0.268 -0.413 +0.006 +0.432 -0.179 +0.094
pre-ReLU (9, 16) [first 14 of 16 columns] =
[CLS] | +0.765 -0.993 -1.055 +0.902 +1.454 +2.195 +0.394 +1.185 -0.517 +0.140 +0.170 +0.026 +0.652 -0.231
p0 | +1.091 -1.256 +0.265 +0.249 +0.357 +1.327 +0.929 +0.037 -1.104 -1.421 +1.023 -0.709 +0.364 +0.236
p1 | +0.764 -1.519 -1.265 -0.755 +1.343 +0.890 +0.397 +0.550 -1.855 +0.017 +1.158 -0.125 +0.799 +0.885
p2 | +0.845 -0.835 +0.094 -0.350 +0.034 -1.019 -0.548 +1.673 -1.660 +0.975 -0.184 +1.517 +0.550 -0.485
p3 | +0.627 -1.334 -0.337 -0.240 +0.806 +1.315 -0.134 +1.542 -0.763 -0.518 +0.578 +0.799 +0.530 -0.118
p4 | -0.165 -1.061 -0.886 -0.374 +1.244 +1.918 +0.100 +0.838 -0.912 -0.865 +0.855 -0.564 +0.895 +0.003
p5 | -0.465 -0.912 -1.665 -0.747 +1.777 +2.131 +0.350 +0.008 -0.651 -0.616 +1.197 -1.120 +0.808 +0.832
p6 | +0.262 -0.626 -0.643 +0.092 +0.412 -0.216 -0.259 +1.245 -2.027 +0.691 -0.134 +0.062 +0.996 -0.666
p7 | +1.356 -1.668 -0.167 -0.001 +0.531 +0.959 +0.401 +1.118 -1.593 -0.699 +0.849 +0.393 +0.528 +0.137
worked element pre-ReLU[0,0] = row 0 of Z1 . column 0 of W1
row 0 of Z1 = [-1.3169 0.7946 -0.6191 -0.8557 -0.9203 0.7243 1.7057 0.4874]
col 0 of W1 = [ 0.2965 -0.3199 -0.3713 -0.2639 -0.3177 -0.1212 0.3423 0.3386]
(-1.3169)*(+0.2965) + (+0.7946)*(-0.3199) + (-0.6191)*(-0.3713) + (-0.8557)*(-0.2639) +
(-0.9203)*(-0.3177) + (+0.7243)*(-0.1212) + (+1.7057)*(+0.3423) + (+0.4874)*(+0.3386)
= +0.764521 | from array: +0.764521 | MATCH: True
ReLU zeroes 61 of 144 hidden activations (42.4%)
==============================================================================
post-ReLU (9, 16) @ W2 (16, 8) = ffn_out (9, 8)
==============================================================================
shape rule: (9 x 16) @ (16 x 8) -> (9 x 8) inner dim 16 cancels
==============================================================================
post-ReLU (9, 16) [first 14 of 16 columns] =
[CLS] | +0.765 +0.000 +0.000 +0.902 +1.454 +2.195 +0.394 +1.185 +0.000 +0.140 +0.170 +0.026 +0.652 +0.000
p0 | +1.091 +0.000 +0.265 +0.249 +0.357 +1.327 +0.929 +0.037 +0.000 +0.000 +1.023 +0.000 +0.364 +0.236
p1 | +0.764 +0.000 +0.000 +0.000 +1.343 +0.890 +0.397 +0.550 +0.000 +0.017 +1.158 +0.000 +0.799 +0.885
p2 | +0.845 +0.000 +0.094 +0.000 +0.034 +0.000 +0.000 +1.673 +0.000 +0.975 +0.000 +1.517 +0.550 +0.000
p3 | +0.627 +0.000 +0.000 +0.000 +0.806 +1.315 +0.000 +1.542 +0.000 +0.000 +0.578 +0.799 +0.530 +0.000
p4 | +0.000 +0.000 +0.000 +0.000 +1.244 +1.918 +0.100 +0.838 +0.000 +0.000 +0.855 +0.000 +0.895 +0.003
p5 | +0.000 +0.000 +0.000 +0.000 +1.777 +2.131 +0.350 +0.008 +0.000 +0.000 +1.197 +0.000 +0.808 +0.832
p6 | +0.262 +0.000 +0.000 +0.092 +0.412 +0.000 +0.000 +1.245 +0.000 +0.691 +0.000 +0.062 +0.996 +0.000
p7 | +1.356 +0.000 +0.000 +0.000 +0.531 +0.959 +0.401 +1.118 +0.000 +0.000 +0.849 +0.393 +0.528 +0.137
W2 (16, 8) =
0 | +0.049 +0.049 -0.298 +0.185 -0.412 -0.361 -0.497 -0.383
1 | -0.027 +0.106 +0.294 -0.393 +0.351 +0.246 -0.091 +0.433
2 | +0.491 -0.295 -0.121 +0.426 +0.222 -0.452 +0.282 +0.328
3 | +0.251 +0.300 +0.325 -0.314 -0.264 +0.134 +0.408 -0.184
4 | +0.088 +0.183 -0.048 +0.214 +0.400 +0.124 +0.040 -0.061
5 | +0.077 -0.145 -0.109 +0.032 -0.433 -0.271 +0.043 -0.068
6 | -0.167 +0.231 +0.194 -0.333 +0.379 -0.005 +0.241 +0.073
7 | +0.498 +0.252 +0.207 +0.279 -0.357 -0.295 +0.214 -0.006
8 | +0.255 -0.397 +0.036 -0.121 -0.043 +0.104 +0.002 +0.040
9 | -0.014 -0.091 +0.272 -0.488 +0.098 +0.066 +0.216 +0.099
10 | +0.327 +0.459 -0.157 -0.273 -0.076 -0.212 +0.115 +0.412
11 | -0.361 -0.399 -0.244 +0.226 +0.093 -0.398 +0.419 +0.290
12 | -0.477 +0.151 +0.271 -0.126 -0.431 -0.423 -0.396 +0.340
13 | +0.411 -0.377 -0.264 -0.334 -0.314 +0.337 -0.168 -0.189
14 | -0.273 +0.108 -0.121 +0.244 -0.294 +0.288 +0.104 -0.386
15 | -0.085 +0.364 +0.423 -0.034 -0.019 +0.418 +0.087 -0.467
ffn_out (9, 8) =
[CLS] | +0.690 +0.850 +0.204 +0.362 -1.613 -1.099 +0.340 -0.544
p0 | +0.257 +0.679 -0.483 -0.130 -1.123 -0.845 -0.037 -0.267
p1 | +0.575 +0.789 -0.489 -0.048 -1.154 -0.568 -0.320 -0.135
p2 | -0.033 -0.116 +0.066 +0.587 -1.055 -1.470 +0.644 +0.230
p3 | +0.437 +0.476 -0.272 +0.879 -1.452 -1.411 +0.367 +0.004
p4 | +0.257 +0.813 -0.081 +0.410 -1.324 -0.901 +0.176 +0.088
p5 | +0.423 +0.558 -0.486 -0.210 -0.985 -0.448 +0.003 +0.085
p6 | -0.032 +0.579 +0.537 +0.200 -1.001 -0.570 +0.054 -0.031
p7 | +0.379 +0.754 -0.453 +0.530 -1.565 -1.278 -0.155 -0.314
worked element ffn_out[0,0] = row 0 of post-ReLU . column 0 of W2
row 0 of post-ReLU = [0.7645 0. 0. 0.9017 1.4536 2.195 0.3945 1.1848 0. 0.1397 0.1698 0.0264 0.6519 0. 0.4703 0. ]
col 0 of W2 = [ 0.0486 -0.0268 0.4909 0.2505 0.0883 0.0775 -0.1672 0.4977 0.2546 -0.0136 0.3268 -0.3609 -0.477 0.4107 -0.2726 -0.0855]
(+0.7645)*(+0.0486) + (+0.0000)*(-0.0268) + (+0.0000)*(+0.4909) + (+0.9017)*(+0.2505) +
(+1.4536)*(+0.0883) + (+2.1950)*(+0.0775) + (+0.3945)*(-0.1672) + (+1.1848)*(+0.4977) +
(+0.0000)*(+0.2546) + (+0.1397)*(-0.0136) + (+0.1698)*(+0.3268) + (+0.0264)*(-0.3609) +
(+0.6519)*(-0.4770) + (+0.0000)*(+0.4107) + (+0.4703)*(-0.2726) + (+0.0000)*(-0.0855)
= +0.690151 | from array: +0.690151 | MATCH: True
Add & Norm 2 -> ENCODER OUTPUT H
==============================================================================
H (9, 8) =
[CLS] | -0.395 +1.311 -0.236 -0.295 -1.827 -0.205 +1.613 +0.033
p0 | +0.806 +0.897 +0.243 -1.745 -1.627 +0.260 +0.427 +0.738
p1 | +0.544 -0.334 +1.070 -0.956 -1.646 -0.097 +1.661 -0.242
p2 | +0.158 -1.918 -0.054 +1.098 -0.730 -0.230 +1.579 +0.097
p3 | -0.998 -0.393 +0.151 +0.555 -1.742 -0.171 +1.495 +1.103
p4 | -1.335 +0.845 +0.776 -0.420 -1.766 +0.468 +1.141 +0.292
p5 | -1.082 +1.079 +1.095 -1.086 -1.216 -0.095 +1.341 -0.036
p6 | +0.097 -0.377 +0.589 +0.317 -1.667 +1.184 +1.195 -1.339
p7 | +0.489 -0.210 +0.375 -0.385 -2.335 +0.137 +1.188 +0.740
H.shape = (9, 8) one contextual vector per position.
Row 0 is the [CLS] summary; rows 1..8 are the contextualised patches.
7. The Reconstruction Head and the Loss
Text BERT ends with a softmax over the vocabulary and a cross-entropy loss. There is no vocabulary for a continuous signal, so the head projects each encoder output back to the patch dimension and the loss is mean squared error:
$$\hat{p}_i = H_i W_r + b_r \in \mathbb{R}^{16}, \qquad \mathcal{L} = \frac{1}{|\mathcal{M}|}\sum_{i \in \mathcal{M}} \big\lVert \hat{p}_i - p_i \big\rVert^2$$
Crucially the sum runs only over the masked positions $\mathcal{M}$. Unmasked positions were handed to the model as input, so reconstructing them would be copying, not learning.
reconstruction head: (8) -> (16) for each of 8 patches
recon.shape = (8, 16)
==============================================================================
H (patches only) (8, 8) @ W_r (8, 16) = reconstruction (8, 16)
==============================================================================
shape rule: (8 x 8) @ (8 x 16) -> (8 x 16) inner dim 8 cancels
==============================================================================
H (patches only) (8, 8) =
p0 | +0.806 +0.897 +0.243 -1.745 -1.627 +0.260 +0.427 +0.738
p1 | +0.544 -0.334 +1.070 -0.956 -1.646 -0.097 +1.661 -0.242
p2 | +0.158 -1.918 -0.054 +1.098 -0.730 -0.230 +1.579 +0.097
p3 | -0.998 -0.393 +0.151 +0.555 -1.742 -0.171 +1.495 +1.103
p4 | -1.335 +0.845 +0.776 -0.420 -1.766 +0.468 +1.141 +0.292
p5 | -1.082 +1.079 +1.095 -1.086 -1.216 -0.095 +1.341 -0.036
p6 | +0.097 -0.377 +0.589 +0.317 -1.667 +1.184 +1.195 -1.339
p7 | +0.489 -0.210 +0.375 -0.385 -2.335 +0.137 +1.188 +0.740
W_r (8, 16) =
0 | +0.413 -0.252 +0.078 -0.334 -0.466 -0.189 +0.280 -0.222 -0.280 -0.287 +0.015 +0.476 -0.041 +0.057 +0.361 +0.035
1 | -0.316 -0.200 -0.190 -0.103 -0.073 +0.300 -0.151 -0.032 +0.125 -0.122 +0.337 +0.087 -0.206 +0.214 +0.028 +0.034
2 | -0.019 -0.003 +0.265 -0.397 -0.166 -0.424 +0.253 -0.228 +0.397 +0.027 +0.301 +0.479 +0.340 +0.367 -0.092 +0.052
3 | -0.246 -0.304 +0.006 +0.095 -0.161 +0.069 +0.387 +0.057 +0.221 +0.305 +0.489 +0.103 +0.307 +0.463 +0.444 -0.359
4 | -0.094 -0.176 -0.413 +0.133 +0.236 +0.348 -0.377 +0.376 +0.143 +0.204 +0.411 +0.125 -0.164 +0.325 -0.137 -0.466
5 | +0.331 -0.155 +0.274 -0.137 +0.361 -0.280 +0.475 +0.280 -0.386 +0.066 +0.485 -0.029 -0.318 -0.015 +0.012 +0.242
6 | +0.199 -0.097 -0.282 +0.145 -0.078 -0.368 +0.401 +0.191 +0.183 +0.323 +0.030 +0.315 -0.002 -0.433 -0.095 -0.003
7 | +0.220 -0.393 -0.360 -0.236 -0.234 +0.232 -0.249 +0.133 -0.006 +0.073 +0.338 -0.096 +0.271 -0.078 -0.146 +0.459
reconstruction (8, 16) =
p0 | +0.960 +0.061 +0.304 -0.989 -0.697 -0.732 +0.201 -0.722 -0.661 -0.990 -0.745 +0.252 -0.287 -1.255 -0.405 +1.856
p1 | +0.944 +0.455 +0.656 -0.571 -0.750 -1.936 +1.404 -0.770 +0.128 -0.203 -1.005 +0.987 +0.349 -1.323 -0.234 +1.034
p2 | +0.729 -0.017 +0.127 +0.410 -0.502 -1.254 +1.520 +0.076 +0.210 +0.875 -0.454 +0.399 +0.923 -0.838 +0.430 -0.132
p3 | +0.219 -0.086 -0.105 +0.114 -0.468 -0.808 +0.933 -0.039 +0.498 +0.705 -0.212 -0.226 +0.980 -1.126 -0.205 +1.032
p4 | -0.118 +0.305 +0.370 -0.191 +0.094 -0.951 +0.805 -0.207 +0.469 +0.232 -0.074 -0.135 +0.233 -0.903 -0.619 +1.240
p5 | -0.199 +0.495 +0.107 -0.233 +0.000 -0.911 +0.350 -0.338 +0.742 +0.052 -0.373 +0.269 +0.079 -0.903 -0.900 +0.969
p6 | +0.560 +0.473 +1.395 -0.092 +0.087 -2.022 +2.359 -0.372 -0.238 +0.156 -0.320 +0.590 -0.096 -0.685 +0.437 +0.353
p7 | +1.019 +0.018 +0.576 -0.660 -0.980 -1.457 +1.352 -0.724 -0.273 -0.252 -0.745 +0.363 +0.570 -1.390 +0.066 +1.625
worked element reconstruction[3,0] = row 3 of H (patches only) . column 0 of W_r
row 3 of H (patches only) = [-0.9984 -0.393 0.1514 0.555 -1.7423 -0.1711 1.4953 1.1031]
col 0 of W_r = [ 0.4127 -0.3155 -0.0189 -0.2461 -0.0936 0.3307 0.1988 0.2202]
(-0.9984)*(+0.4127) + (-0.3930)*(-0.3155) + (+0.1514)*(-0.0189) + (+0.5550)*(-0.2461) +
(-1.7423)*(-0.0936) + (-0.1711)*(+0.3307) + (+1.4953)*(+0.1988) + (+1.1031)*(+0.2202)
= +0.219234 | from array: +0.219234 | MATCH: True
LOSS - computed on masked patches only
patch masked MSE counts toward loss?
--------------------------------------------------
p0 - 0.775568 no (input, not a target)
p1 - 0.935136 no (input, not a target)
p2 - 0.880113 no (input, not a target)
p3 YES 1.362692 YES
p4 - 0.606813 no (input, not a target)
p5 - 0.589623 no (input, not a target)
p6 - 1.352232 no (input, not a target)
p7 YES 1.349081 YES
--------------------------------------------------
masked patches: ['p3', 'p7']
their MSEs : [1.3627 1.3491]
loss = mean = (1.362692 + 1.349081) / 2
= 2.711772 / 2 = 1.355886
worked example, patch p3:
predicted[:4] = [ 0.2192 -0.0855 -0.1055 0.1145]
true[:4] = [-0.8272 -0.3474 0.638 1.6286]
diff[:4] = [ 1.0465 0.2619 -0.7435 -1.5141]
sum of squares over 16 values = 21.803068
/ 16 = 1.362692
from array = 1.362692 MATCH: True
baseline (predict the mean of the masked patches): 0.492816
model loss with RANDOM, untrained weights : 1.355886
The weights here were never trained, so there is nothing to celebrate in
that number. What has been shown is the mechanism, not a result.
With random weights the reconstruction is unrelated to the target - as it must be. Training is what closes that gap, and the loss above is the signal that would drive it.
8. What [CLS] Is For
The reconstruction head is thrown away after pre-training. What survives is the encoder,
and the usual way to attach a classifier is through the [CLS] position:
$$\hat{y} = \sigma\big(H_{\texttt{[CLS]}} \, w_c + b_c\big)$$
[CLS] carries no signal of its own, so everything in it arrived through attention from
the patches. That is what makes it a whole-recording summary — useful for "does this
epoch contain a seizure?".
H[CLS] = [-0.3946 1.3115 -0.2359 -0.2945 -1.8269 -0.2055 1.6127 0.0333]
logit = H_CLS . w_c + b_c
= (-0.3946)*(-0.0796) + (+1.3115)*(+0.4269) + (-0.2359)*(+0.0815) + (-0.2945)*(+0.1528) +
(-1.8269)*(+0.1913) + (-0.2055)*(+0.0470) + (+1.6127)*(+0.1838) + (+0.0333)*(-0.0418)
= +0.463005
P(seizure) = sigmoid(+0.463005) = 1 / (1 + exp(-0.463005)) = 0.613727
where [CLS] got its content - its attention row (head 0):
attends to weight
------------------------
[CLS] 0.1081 ######
p0 0.1568 #########
p1 0.1121 ######
p2 0.0689 ####
p3 0.1127 ######
p4 0.1253 #######
p5 0.1307 #######
p6 0.0746 ####
p7 0.1107 ######
------------------------
sum = 1.000000
[CLS] began as a fixed learned vector identical for every recording.
Everything that distinguishes one recording's [CLS] from another's came
through these attention weights.
9. Summary
| Stage | Operation | Shape |
|---|---|---|
| Raw EEG | 2 channels x 64 samples | $(2, 64)$ |
| Patching | 8 patches of 8 samples, channels flattened | $(8, 16)$ |
| Patch embedding | $\text{patch}\,W_e + b_e$ | $(8, 8)$ |
Add [CLS] + PE |
prepend and add | $(9, 8)$ |
| Masking | 2 positions → [MASK] vector |
$(9, 8)$ |
| Self-attention | $\mathrm{softmax}(QK^\top/\sqrt{d_k})V$, no mask | $(9, 9)$ weights |
| Encoder output $H$ | attention + FFN, each with Add & Norm | $(9, 8)$ |
| Reconstruction | $H_{1:}W_r + b_r$ | $(8, 16)$ |
| Loss | MSE on masked positions only | scalar |
The one idea: remove the causal mask and the model stops being a generator and becomes an encoder. Prediction of the next item is replaced by reconstruction of a missing one, which is a task that requires understanding context from both directions.
Why this transfers to EEG so directly: BERT never actually required words. It required a sequence of vectors, a way to corrupt some of them, and a loss comparing the reconstruction to the original. Patching supplies the vectors; MSE supplies the loss. The attention mathematics is untouched.
What is deliberately left out: multiple stacked encoder layers (this is one block), learned rather than sinusoidal positional embeddings, the 80/10/10 corruption applied stochastically, GELU instead of ReLU, dropout, next-sentence prediction, and any actual training — every weight here is random.
Compared with the autoregressive Transformer walkthrough:
| Autoregressive (GPT-style) | BERT (this notebook) | |
|---|---|---|
| Mask | causal, upper triangle $= -\infty$ | none |
| Context | left only | both directions |
| Task | predict the next token | reconstruct masked inputs |
| Loss | cross-entropy over vocabulary | MSE over the masked patches |
| Stacks | encoder + decoder | encoder only |
| Good at | generating sequences | representing them |