techtree / Glue math
Nonlinearity (ReLU)
Clip negatives to zero, entry by entry; the bend that makes layers useful.
Transcript
Here is a very simple function. If a number is negative, make it zero. Otherwise, leave it alone.
Three stays three. Minus two becomes zero. Zero stays zero.
Draw it as a graph. Flat along zero on the left, and a straight ramp on the right.
Two straight pieces, joined by one bend. So the whole thing is not a straight line.
It is called ReLU.
On a vector, it works on each entry by itself. Two stays two. Minus one becomes zero. One half stays. Minus three becomes zero.
Negatives become zero. Everything else passes through.
Needs
- Vectors video
A vector is a list of numbers, and also an arrow.
Leads to
- A neural network layer video
Multiply by a matrix, add a bias, apply a nonlinearity.