techtree / Building blocks
MLP
Two layers in a row; widen, bend, narrow.
Transcript
An MLP is two layers in a row. A vector goes in.
The first layer makes it wider: two entries become four. Then ReLU zeroes the negative one.
The second layer makes it narrow again, back to two entries. This one usually has no ReLU at the end.
MLP stands for multi-layer perceptron, an old name. Widen, bend, narrow.
What do the bends buy you? Feed in a single number, and plot the single number that comes out.
Each neuron in the wide middle can add one bend to the line. Two neurons, two bends. Three, three.
With enough of them, the output can follow almost any curve.
That is the job of the wide middle.
Needs
- A neural network layer video
Multiply by a matrix, add a bias, apply a nonlinearity.
Everything this builds on (7 topics, in order)
Leads to
- The transformer block video
Attention then MLP, each wrapped in normalization and a residual.