techtree / Attention
Queries, keys, and values
Three matrices turn each token vector into a query, a key, and a value.
Transcript
Here is one token's vector. The model keeps three matrices for this step.
Multiply the vector by the first, and you get a query. By the second, a key. By the third, a value.
Every token goes through the same three matrices, so every token gets its own query, key, and value.
The query says what this token is looking for. The key says what it has to offer. The value is what it hands over.
A query and a key match when they point the same way, so their dot product is large.
One vector in. Three vectors out: a query, a key, and a value.
Needs
- Embeddings video
Each token id is looked up as a vector; similar meanings point in similar directions. - Matrix times vector video
Each output entry is a dot product of one row with the vector.
Everything this builds on (8 topics, in order)
Leads to
- Attention scores video
Dot each query with every key, then softmax into weights.