techtree / Attention

Causal masking

Tokens may only look backward, never at the future.

46 seconds

later token: score→−∞,e−∞=0\text{later token: score} \to -\infty, \quad e^{-\infty} = 0
Transcript

A model that writes text writes one token at a time. While it works on a token, the tokens after it do not exist yet.

So each token may look at itself and the tokens before it. Never ahead.

Here are the scores, one row for each token. Every score that looks ahead is replaced with minus infinity.

Then softmax. e to the minus infinity is zero, so those weights vanish. The others still add up to one.

The first token can only look at itself. The last can look at everything.

The weights form a triangle. This is called a causal mask. Each token sees the past, and only the past.

Needs

  • Self-attention video
    Each token's output is a weighted average of the value vectors.
Everything this builds on (12 topics, in order)
  1. Vectors
  2. Tokens
  3. Adding and scaling vectors
  4. Dot product
  5. Matrices
  6. Exp and softmax
  7. Length, angle, and cosine similarity
  8. Matrix times vector
  9. Embeddings
  10. Queries, keys, and values
  11. Attention scores
  12. Self-attention

Leads to