techtree / Transformer

What is a transformer?

The whole machine, end to end.

46 seconds

text→tokens→vectors→blocks→softmax→next token\text{text} \to \text{tokens} \to \text{vectors} \to \text{blocks} \to \text{softmax} \to \text{next token}
Transcript

A transformer takes in text, chops it into tokens, and looks up a vector for each one, with its position added.

The vectors rise through a stack of blocks. In each block, attention lets the tokens share information, and an MLP works on each token by itself.

At the top, the last vector becomes a score for every token in the vocabulary. Softmax makes them probabilities, and one token is picked.

It is added to the text, and the whole machine runs again.

All of its skill lives in its weights: the embedding table, the attention matrices, and the MLPs.

Text in. One token out. Again and again. That is a transformer.

Needs

Everything this builds on (23 topics, in order)
  1. Vectors
  2. Tokens
  3. Adding and scaling vectors
  4. Dot product
  5. Matrices
  6. Exp and softmax
  7. Nonlinearity (ReLU)
  8. Mean, variance, and normalization
  9. Length, angle, and cosine similarity
  10. Matrix times vector
  11. Embeddings
  12. A neural network layer
  13. Position information
  14. Queries, keys, and values
  15. MLP
  16. Residual connections
  17. Attention scores
  18. Self-attention
  19. Causal masking
  20. Multi-head attention
  21. The transformer block
  22. Stacking blocks
  23. Next-token prediction and sampling

Leads to

This is the top of the tree.