techtree / Building blocks

Residual connections

Add a block's output back to its input, so each block makes an edit.

40 seconds

\mathbf{y} = \mathbf{x} + \text{block}(\mathbf{x})

Transcript

Here is a block: a layer that takes in a vector and gives one back. On its own, its output replaces the input.

A residual connection adds a second path, around the block. The input is added to the block's output.

Here is the input, as an arrow. The block's output is a small arrow. Add it tip to tail, and you get the input, nudged.

So the block does not rewrite the vector. It makes an edit.

If the block gives back all zeros, the input passes through unchanged. Doing nothing is easy.

The input always has a clear way through.

Needs

A neural network layer video
Multiply by a matrix, add a bias, apply a nonlinearity.
Adding and scaling vectors video
Add vectors entry by entry (tip to tail); scale by multiplying every entry.

Everything this builds on (7 topics, in order)

The transformer block video
Attention then MLP, each wrapped in normalization and a residual.