techtree / Attention

Multi-head attention

Run several small attentions side by side and combine them.

54 seconds

out=WO[head1⋮headh]\text{out} = W_O \begin{bmatrix} \text{head}_1 \\ \vdots \\ \text{head}_h \end{bmatrix}
Transcript

In one attention, each token gets one set of weights, so it blends in one way. But a word relates to the others in more than one way.

Here, draws looks to robot, to find who is drawing. It also looks to circles, to find what is drawn.

So the model runs several attentions side by side. Each one is called a head. Each head has its own query, key, and value matrices, so each can look for something different.

The heads work with shorter vectors, so together they cost about as much as one big attention.

Each head gives each token a short new vector. Set them end to end, into one long vector. Then one more matrix mixes them together.

Real models use a dozen heads or more. Several ways of looking, combined into one vector.

Needs

  • Self-attention video
    Each token's output is a weighted average of the value vectors.
  • Matrix times vector video
    Each output entry is a dot product of one row with the vector.
Everything this builds on (12 topics, in order)
  1. Vectors
  2. Tokens
  3. Adding and scaling vectors
  4. Dot product
  5. Matrices
  6. Exp and softmax
  7. Length, angle, and cosine similarity
  8. Matrix times vector
  9. Embeddings
  10. Queries, keys, and values
  11. Attention scores
  12. Self-attention

Leads to