techtree / Glue math

Mean, variance, and normalization

Shift and scale a vector so its entries have mean 0 and spread 1.

43 seconds

\text{normalized entry} = \frac{\text{entry} - \text{mean}}{\sqrt{\text{variance}}}

Transcript

Here is a vector: three, five, six, seven, nine. Draw each entry as a bar.

Its mean is the average: add the entries up, and divide by how many there are. Thirty divided by five is six.

Subtract six from every entry. Now the entries are centered on zero.

How spread out are they? Square each entry, and average the squares. That average, four, is called the variance.

Its square root, two, is the typical distance from zero.

Divide every entry by two. Now the mean is zero, and the spread is one.

The pattern of the bars stays the same. Only the center and the size change.

Needs

Adding and scaling vectors video
Add vectors entry by entry (tip to tail); scale by multiplying every entry.

Everything this builds on (2 topics, in order)

The transformer block video
Attention then MLP, each wrapped in normalization and a residual.