techtree / Transformer

Next-token prediction and sampling

Turn the last vector into a score per token, softmax, and pick one.

51 seconds

probabilities=softmax(WxlastT)\text{probabilities} = \text{softmax}\left( \frac{W \mathbf{x}_{\text{last}}}{T} \right)
Transcript

After the last block, take the vector at the last position. Multiply it by one more matrix, with one row for every token in the vocabulary. Often this is the embedding table itself, reused.

Out comes one score for every token. Softmax turns the scores into probabilities: how likely each token is to come next.

Now pick one. Always taking the top one is called greedy. Or draw at random, following the probabilities. That is sampling.

A setting called temperature divides the scores first. Low temperature makes the top token more likely. High temperature spreads the chances out.

The chosen token is added to the end of the text, and the whole model runs again, for the token after that.

One token at a time, that is how the text gets written.

Needs

  • Stacking blocks video
    The same block design, repeated many times, each with its own weights.
  • Exp and softmax video
    Turn any list of scores into positive weights that add up to one.
  • Matrix times vector video
    Each output entry is a dot product of one row with the vector.
Everything this builds on (22 topics, in order)
  1. Vectors
  2. Tokens
  3. Adding and scaling vectors
  4. Dot product
  5. Matrices
  6. Exp and softmax
  7. Nonlinearity (ReLU)
  8. Mean, variance, and normalization
  9. Length, angle, and cosine similarity
  10. Matrix times vector
  11. Embeddings
  12. A neural network layer
  13. Position information
  14. Queries, keys, and values
  15. MLP
  16. Residual connections
  17. Attention scores
  18. Self-attention
  19. Causal masking
  20. Multi-head attention
  21. The transformer block
  22. Stacking blocks

Leads to