techtree / Building blocks

A neural network layer

Multiply by a matrix, add a bias, apply a nonlinearity.

47 seconds

y=ReLU(Wx+b)\mathbf{y} = \text{ReLU}(W \mathbf{x} + \mathbf{b})
Transcript

A layer takes in a vector. First, it multiplies it by a matrix: two numbers in, three numbers out.

Then it adds another vector, called the bias. That shifts each entry up or down.

Last, it applies ReLU to each entry. The negative one becomes zero. The others pass through.

Each output entry, with its row of the matrix and its bias, is called a neuron.

The numbers in the matrix and the bias are the layer's weights. In a real network, they are tuned, not written by hand.

Real layers are much bigger, with hundreds of entries in and out. The steps are the same.

Multiply by a matrix, add a bias, apply ReLU. That is one layer.

Needs

Everything this builds on (6 topics, in order)
  1. Vectors
  2. Adding and scaling vectors
  3. Dot product
  4. Matrices
  5. Nonlinearity (ReLU)
  6. Matrix times vector

Leads to

  • MLP video
    Two layers in a row; widen, bend, narrow.
  • Residual connections video
    Add a block's output back to its input, so each block makes an edit.