NN - 24 - Activations - Part 2: ReLU Variants

Описание: In this video we will look at different ReLU variants: Leaky ReLU (LReLU), Parameterized Leaky ReLU (PReLU), Exponential Linear Unit (ELU), Scaled Exponential Linear Units (SELU), Continuously Differentiable Exponential Linear Units (CELU), Gaussian Error Linear Units (GELU), Sigmoid Linear Unit (SiLU), Swish, Mish, Randomized Leaky ReLU (RReLU), Softplus, Log-Sigmoid. We will also discuss shortly the Soft-sign function.

Papers:
Rectifier nonlinearities improve neural network acoustic models, Maas et al. 2013
Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification, He et al. 2015
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), Clevert et al., 2016
Self-Normalizing Neural Networks, Klambauer et al., 2017
Continuously Differentiable Exponential Linear Units, Barron, 2017
Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units, Hendrycks & Gimpel 2016
Gaussian Error Linear Units (GELUs), Hendrycks & Gimpel 2020
Searching for Activation Functions, Ramachandran et al., 2017
Mish: A Self Regularized Non-Monotonic Activation Function, Misra 2020
Rectified Linear Units Improve Restricted Boltzmann Machines, Nair & Hinton, 2010

NN Playlist: https://bit.ly/3PvvYSF

Become a member and get full access to this online course:
https://meerkatstatistics.com/courses...

** 🎉 Special YouTube 60% Discount on Yearly Plan – valid for the 1st 100 subscribers; Voucher code: First100 🎉 **

"NN with Python" Course Outline:
Intro
Administration
Intro - Long
Notebook - Intro to Python
Notebook - Intro to PyTorch
Comparison to other methods
Linear Regression vs. Neural Network
Logistic Regression vs. Neural Network
GLM vs. Neural Network
Expressivity / Capacity
Hidden Layers: 0 vs. 1 vs. 2+
Training
Backpropagation - Part 1
Backpropagation - Part 2
Implement a NN in NumPy
Notebook - Implementation redo: Classes instead of Functions (NumPy)
Classification - Softmax and Cross Entropy - Theory
Classification - Softmax and Cross Entropy - Derivatives
Notebook - Implementing Classification (NumPy)
Autodiff
Automatic Differentiation
Forward vs. Reverse mode
Symmetries in Weight Space
Tanh & Permutation Symmetries
Notebook - Tanh, Permutation, ReLU symmetries
Generalization
Generalization and the Bias-Variance Trade-Off
Generalization Code
L2 Regularization / Weight Decay
DropOut regularization
Notebook - DropOut (PyTorch)
Notebook - DropOut (NumPy)
Notebook - Early Stopping
Improved Training
Weight Initialization - Part 1: What NOT to do
Notebook - Weight Initialization 1
Weight Initialization - Part 2: What to do
Notebook - Weight Initialization 2
Notebook - TensorBoard
Learning Rate Decay
Notebook - Input Normalization
Batch Normalization - Part 1: Theory
Batch Normalization - Part 2: Derivatives
Notebook - BatchNorm (PyTorch)
Notebook - BatchNorm (NumPy)
Activation Functions
Classical Activations
ReLU Variants
Optimizers
SGD Variants: Momentum, NAG, AdaGrad, RMSprop, AdaDelta, Adam, AdaMax, Nadam - Part 1: Theory
SGD Variants: Momentum, NAG, AdaGrad, RMSprop, AdaDelta, Adam, AdaMax, Nadam - Part 2: Code
Auto Encoders
Variational Auto Encoders

If you’re looking for statistical consultation, work on interesting projects, or training workshop, visit my website https://meerkatstatistics.com/ or contact me directly at [email protected]

~~~~~ SUPPORT ~~~~~
Paypal me: https://paypal.me/MeerkatStatistics
~~~~~~~~~~~~~~~~~

Intro/Outro Music: Dreamer - by Johny Grimes
   • Johny Grimes - Dreamer