Here are the pictures of two different problems.
Two classes. We want to use a neural network to solve these problems.
What's the difference between them?
1 of 8
Two classes. We want to use a neural network to solve these problems.
What's the difference between them?
1 of 8
You can separate both classes from the first problem using a line.
This is a "linearly-separable" problem.
2 of 8
This is a "linearly-separable" problem.
2 of 8
You can't use a line to separate the two classes on the second problem.
This is a much more complex problem.
3 of 8
This is a much more complex problem.
3 of 8
Here is a neural network with 2 hidden layers of 4 neurons.
In just a few iterations, the network finds the correct solution for the first problem.
Notice how the network uses a single dividing line in the output. That's all it can do with linear activations.
4 of 8
In just a few iterations, the network finds the correct solution for the first problem.
Notice how the network uses a single dividing line in the output. That's all it can do with linear activations.
4 of 8
I tried the same network on the second problem.
It doesn't work.
If we want to solve this problem, we need to allow this network to find non-linear solutions.
5 of 8
It doesn't work.
If we want to solve this problem, we need to allow this network to find non-linear solutions.
5 of 8
I changed the Activation from "Linear" to ReLU.
This introduces the non-linearity we need for the network to become more powerful.
Same network structure, with just a different activation.
And now we can solve the problem!
6 of 8
This introduces the non-linearity we need for the network to become more powerful.
Same network structure, with just a different activation.
And now we can solve the problem!
6 of 8
Main takeaway:
A neural network without non-linear activation functions can only find linear separation boundaries.
The pictures on this thread come from the TensorFlow Playground, which you can try here: playground.tensorflow.org.
7 of 8
A neural network without non-linear activation functions can only find linear separation boundaries.
The pictures on this thread come from the TensorFlow Playground, which you can try here: playground.tensorflow.org.
7 of 8
Every week, I break down machine learning concepts to give you ideas on applying them in real-life situations.
Follow me @svpino to ensure you don't miss what's coming next.
8 of 8
Follow me @svpino to ensure you don't miss what's coming next.
8 of 8
Loading suggestions...