Gradient descent, vanishing gradients, exploding gradients...
Gradients are everywhere in ML.
But what does Gradient mean?
Let me explain! ๐งต
1/10
Gradients are everywhere in ML.
But what does Gradient mean?
Let me explain! ๐งต
1/10
Gradients are really important in Machine Learning.
We use gradient information in deep learning and many other algorithms to train models and optimize them.
To understand gradients first we need to know derivatives.
I know it sounds horrible, but stay with me!
2/10
We use gradient information in deep learning and many other algorithms to train models and optimize them.
To understand gradients first we need to know derivatives.
I know it sounds horrible, but stay with me!
2/10
In mathematics, the derivative is the rate of change of a function's output with respect to the input.
The good news?
We can visualize it!
Visually the derivative is the slope of the tangent line to the graph of the function at a point.
3/10
The good news?
We can visualize it!
Visually the derivative is the slope of the tangent line to the graph of the function at a point.
3/10
Consider these points:
At A the slope of tangent is steep. A small change in X will result in a big change in Y.
At B the slope is flatter. A small change in X will result in only a small change in Y.
4/10
At A the slope of tangent is steep. A small change in X will result in a big change in Y.
At B the slope is flatter. A small change in X will result in only a small change in Y.
4/10
How to interpret a derivative?
A negative derivative indicates a decrease.
A derivative of 0 indicates no change.
A positive derivative indicates an increase.
(Reading from left to right)
5/10
A negative derivative indicates a decrease.
A derivative of 0 indicates no change.
A positive derivative indicates an increase.
(Reading from left to right)
5/10
What is a gradient?
A gradient is a derivative of a function with more input variables.
So a gradient is basically the same as a derivative but with more dimensions.
In ML, we typically have a set of parameters that define a model, so we use gradients.
6/10
A gradient is a derivative of a function with more input variables.
So a gradient is basically the same as a derivative but with more dimensions.
In ML, we typically have a set of parameters that define a model, so we use gradients.
6/10
Why knowing the derivative/gradient is good?
Because we can optimize.
The derivative can tell how to change the input in order to increase or decrease the output, so we can get closer to minimum or maximum of a function.
Go back to the A-B example.
7/10
Because we can optimize.
The derivative can tell how to change the input in order to increase or decrease the output, so we can get closer to minimum or maximum of a function.
Go back to the A-B example.
7/10
At A since the slope is steep we are far from the minimum, we know that we need to change X dramatically.
At B since the slope is flatter, we know that we are closer to the minimum.
Looking for the minimum. Sounds familiar?
This is exactly what gradient descent does!
8/10
At B since the slope is flatter, we know that we are closer to the minimum.
Looking for the minimum. Sounds familiar?
This is exactly what gradient descent does!
8/10
In gradient descent, we can determine how changing each parameter will affect the cost function.
At every point, we know how to tweak the inputs, so the cost function will get closer to the minimum (where the derivative is 0).
9/10
At every point, we know how to tweak the inputs, so the cost function will get closer to the minimum (where the derivative is 0).
9/10
That's it for today.
I hope you've found this thread helpful.
Like/Retweet the first tweet below for support and follow @levikul09 for more Data Science threads.
Thanks ๐
10/10
I hope you've found this thread helpful.
Like/Retweet the first tweet below for support and follow @levikul09 for more Data Science threads.
Thanks ๐
10/10
Loading suggestions...