techtree / Transformer
The transformer block
Attention then MLP, each wrapped in normalization and a residual.
Transcript
A transformer block puts the pieces together. In come the token vectors, with their positions added.
First, normalize each vector. Then multi-head attention, with the causal mask, lets the tokens share information. Its result is added back to the input.
Normalize again. Then the MLP works on each token's vector by itself, and its result is added back too.
Attention moves information between tokens. The MLP works within each token.
Out come the same number of vectors, each the same size as before, each one edited.
Normalize, attend, add. Normalize, MLP, add. That is one block.
Needs
- Multi-head attention video
Run several small attentions side by side and combine them. - Causal masking video
Tokens may only look backward, never at the future. - MLP video
Two layers in a row; widen, bend, narrow. - Residual connections video
Add a block's output back to its input, so each block makes an edit. - Mean, variance, and normalization video
Shift and scale a vector so its entries have mean 0 and spread 1. - Position information video
Add a position vector to each token vector so order is not lost.
Everything this builds on (20 topics, in order)
- Vectors
- Tokens
- Adding and scaling vectors
- Dot product
- Matrices
- Exp and softmax
- Nonlinearity (ReLU)
- Mean, variance, and normalization
- Length, angle, and cosine similarity
- Matrix times vector
- Embeddings
- A neural network layer
- Position information
- Queries, keys, and values
- MLP
- Residual connections
- Attention scores
- Self-attention
- Causal masking
- Multi-head attention
Leads to
- Stacking blocks video
The same block design, repeated many times, each with its own weights.