5 facts about Tanh.
Tanh is an activation function used in complex Neural Networks.
Here are some facts you must know about it.
๐งต
Tanh is an activation function used in complex Neural Networks.
Here are some facts you must know about it.
๐งต
1. Mathematical representation
Tanh is a relatively complex function compared to others like ReLU.
For this reason, it's computationally more expensive and it's usually used in hidden layers.
Tanh is a relatively complex function compared to others like ReLU.
For this reason, it's computationally more expensive and it's usually used in hidden layers.
2. Range
The output of the tanh function lies between -1 and 1.
Other functions like sigmoid allow only positive values, while tanh can be negative.
This can be an advantage in some neural networks.
The output of the tanh function lies between -1 and 1.
Other functions like sigmoid allow only positive values, while tanh can be negative.
This can be an advantage in some neural networks.
3. Zero centered
The output of the function averages around 0. Or it's 'symmetric around the origin'.
Thanks to the symmetry it can balance between the negative and positive values, resulting in better flexibility.
The output of the function averages around 0. Or it's 'symmetric around the origin'.
Thanks to the symmetry it can balance between the negative and positive values, resulting in better flexibility.
4. Saturation
When the input values are very high or very low, the function saturates.
Saturation means that the graph 'flattens out'.
This can lead to the vanishing gradient problem since the gradients are very small in the saturated region.
When the input values are very high or very low, the function saturates.
Saturation means that the graph 'flattens out'.
This can lead to the vanishing gradient problem since the gradients are very small in the saturated region.
5. Shape
While saturation is a problem at extreme values, the S-shape in the middle provides smooth updates in the network.
The changes will be consistent and predictable.
While saturation is a problem at extreme values, the S-shape in the middle provides smooth updates in the network.
The changes will be consistent and predictable.
That's it for today.
I hope you've found this thread helpful.
Like/Retweet the first tweet below for support and follow @levikul09 for more Data Science threads.
Thanks ๐
I hope you've found this thread helpful.
Like/Retweet the first tweet below for support and follow @levikul09 for more Data Science threads.
Thanks ๐
If you haven't already, join our newsletter DSBoost.
We share:
โข Interviews
โข Podcast notes
โข Learning resources
โข Interesting collections of content
dsboost.dev
We share:
โข Interviews
โข Podcast notes
โข Learning resources
โข Interesting collections of content
dsboost.dev
Loading suggestions...