techtree / Building blocks

Position information

Add a position vector to each token vector so order is not lost.

53 seconds

inputi=embedding(tokeni)+pi\text{input}_i = \text{embedding}(\text{token}_i) + \mathbf{p}_i
Transcript

A token gets the same vector wherever it appears. Dog bites man. Man bites dog. The same three vectors. Nothing in them says which word came first.

So each position in the sentence gets a vector of its own. Add it to the vector of the token sitting there.

Here is the vector for dog. In the first position, it gets one nudge. In the third position, a different one. Same word, different place, different vector.

The position vectors can be kept in a table, like the embeddings. Or they can be built from waves, some fast, some slow, so every position gets its own pattern.

Now each vector says what the token is, and where it is.

Needs

  • Embeddings video
    Each token id is looked up as a vector; similar meanings point in similar directions.
  • Adding and scaling vectors video
    Add vectors entry by entry (tip to tail); scale by multiplying every entry.
Everything this builds on (6 topics, in order)
  1. Vectors
  2. Tokens
  3. Adding and scaling vectors
  4. Dot product
  5. Length, angle, and cosine similarity
  6. Embeddings

Leads to