techtree / Building blocks

Embeddings

Each token id is looked up as a vector; similar meanings point in similar directions.

58 seconds

embedding=row id of the table\text{embedding} = \text{row } \textit{id} \text{ of the table}
Transcript

A sentence has become a list of token ids. But an id is only a name tag. The number itself means nothing.

So the model keeps a table, with one row for each token in the vocabulary. Each row holds a list of numbers: a vector.

To embed a token, use its id to find its row. That vector is the token's embedding.

Each token in the sentence gets its own vector.

Nobody writes these numbers by hand. They are tuned, so that tokens with similar meanings get vectors that point in similar directions.

Real embeddings have hundreds of entries. These made-up ones have just two, so we can draw them.

Cat and dog point almost the same way. Their cosine similarity is point nine eight. Cat and truck point far apart: point two one.

The id says which token it is. The embedding says what it is like.

Needs

Everything this builds on (5 topics, in order)
  1. Vectors
  2. Tokens
  3. Adding and scaling vectors
  4. Dot product
  5. Length, angle, and cosine similarity

Leads to