techtree / Attention

Attention scores

Dot each query with every key, then softmax into weights.

46 seconds

weights=softmax(q⋅kjentries)\text{weights} = \text{softmax}\left( \frac{\mathbf{q} \cdot \mathbf{k}_j}{\sqrt{\text{entries}}} \right)
Transcript

Take the query of one token: draws. Line it up against the key of every token in the sentence.

Dot the query with each key: one, four, two. A bigger number means a better match.

Then divide each by the square root of the number of entries. Here that is two. This keeps the scores from getting too large.

Softmax turns the scores into weights. They are all positive, and they add up to one.

Draws pays the most attention to plotter.

Every token does the same, with its own query. Each row holds one token's weights, and every row adds up to one.

A query, held up against every key, decides where a token looks.

Needs

  • Queries, keys, and values video
    Three matrices turn each token vector into a query, a key, and a value.
  • Dot product video
    Multiply matching entries and add them up; one number that measures agreement.
  • Exp and softmax video
    Turn any list of scores into positive weights that add up to one.
Everything this builds on (10 topics, in order)
  1. Vectors
  2. Tokens
  3. Adding and scaling vectors
  4. Dot product
  5. Matrices
  6. Exp and softmax
  7. Length, angle, and cosine similarity
  8. Matrix times vector
  9. Embeddings
  10. Queries, keys, and values

Leads to

  • Self-attention video
    Each token's output is a weighted average of the value vectors.