techtree

A map of what you need to know to understand a transformer, the machine inside modern AI. Each topic gets one short video and builds only on the topics above it.

Assumes: arithmetic, basic algebra, functions and their graphs, exponents, square roots. 26 of 26 videos made so far.

has a video not yet

Linear algebra

Vectors video
A vector is a list of numbers, and also an arrow.
Adding and scaling vectors video
Add vectors entry by entry (tip to tail); scale by multiplying every entry.
Dot product video
Multiply matching entries and add them up; one number that measures agreement.
Matrices video
A matrix is a grid of numbers; a stack of vectors.
Length, angle, and cosine similarity video
Length comes from the dot product with itself; dividing out lengths leaves the angle.
Matrix times vector video
Each output entry is a dot product of one row with the vector.
Transpose video
Flip a matrix so rows become columns.
Matrix times matrix video
Multiply a matrix by many vectors at once.

Glue math

Exp and softmax video
Turn any list of scores into positive weights that add up to one.
Nonlinearity (ReLU) video
Clip negatives to zero, entry by entry; the bend that makes layers useful.
Mean, variance, and normalization video
Shift and scale a vector so its entries have mean 0 and spread 1.

Building blocks

Tokens video
Text is chopped into pieces, and each piece gets an id number.
Embeddings video
Each token id is looked up as a vector; similar meanings point in similar directions.
A neural network layer video
Multiply by a matrix, add a bias, apply a nonlinearity.
Position information video
Add a position vector to each token vector so order is not lost.
MLP video
Two layers in a row; widen, bend, narrow.
Residual connections video
Add a block's output back to its input, so each block makes an edit.

Attention

Queries, keys, and values video
Three matrices turn each token vector into a query, a key, and a value.
Attention scores video
Dot each query with every key, then softmax into weights.
Self-attention video
Each token's output is a weighted average of the value vectors.
Causal masking video
Tokens may only look backward, never at the future.
Multi-head attention video
Run several small attentions side by side and combine them.

Transformer

The transformer block video
Attention then MLP, each wrapped in normalization and a residual.
Stacking blocks video
The same block design, repeated many times, each with its own weights.
Next-token prediction and sampling video
Turn the last vector into a score per token, softmax, and pick one.
What is a transformer? video
The whole machine, end to end.