techtree / Transformer

Stacking blocks

The same block design, repeated many times, each with its own weights.

43 seconds

x→block1→block2→⋯→blockN\mathbf{x} \to \text{block}_1 \to \text{block}_2 \to \cdots \to \text{block}_N
Transcript

One block makes one round of edits to the token vectors. A transformer stacks many blocks, one on top of the next. What comes out of one block goes into the next.

Every block has the same design, but its own weights: its own attention matrices, and its own MLP.

The residual stream runs up through all of them. Each block reads the vectors, and adds its edit.

Small models have about a dozen blocks. The largest have around a hundred.

Early blocks tend to pick up simple patterns, like which words sit nearby. Later blocks build on those, toward meaning.

One design, stacked many times, each copy with its own weights.

Needs

Everything this builds on (21 topics, in order)
  1. Vectors
  2. Tokens
  3. Adding and scaling vectors
  4. Dot product
  5. Matrices
  6. Exp and softmax
  7. Nonlinearity (ReLU)
  8. Mean, variance, and normalization
  9. Length, angle, and cosine similarity
  10. Matrix times vector
  11. Embeddings
  12. A neural network layer
  13. Position information
  14. Queries, keys, and values
  15. MLP
  16. Residual connections
  17. Attention scores
  18. Self-attention
  19. Causal masking
  20. Multi-head attention
  21. The transformer block

Leads to