DataTerminology.comNEW
Also known as: Kaiming Initialization
A neural network weight initialization tailored for ReLU activations that preserves variance across layers to stabilize training.
DESCRIPTION
He initialization, also known as Kaiming initialization, sets layer weights with variance scaled to the number of input units for rectified linear unit activations. By maintaining signal variance across depth, it mitigates vanishing or exploding gradients in deep networks. It is commonly used with ReLU and variants like Leaky ReLU. Implementations are available as normal and uniform distributions in libraries such as PyTorch and TensorFlow. This method helps speed convergence and improve model stability during training.
ALTERNATIVE DEFINITIONS
arXiv (He et al.)
We propose a new initialization method that specifically accounts for the rectifier nonlinearity, leading to improved convergence properties in deep networks.
Introduces initialization theoretically derived for rectifier nonlinearities with empirical improvements.
Stanford CS231n
He initialization: set weights to random values with variance 2/n where n is the number of inputs; recommended for layers with ReLU activations.
Provides practical formula and guidance for ReLU-based layers.
RELATED TERMS
ReLU Deep Learning Batch Normalization ResNet Layer Normalization Gradient Vanishing Gradient Exploding
COMPETING TERMS
Xavier Initialization LeCun Initialization
RELATIONS
- Broader: Initialization
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - Rectifier (neural networks)
- PyTorch - Kaiming normal initializer for ReLU
- TensorFlow - He normal initializer in Keras
TOP QUESTIONS
Why is He Initialization important for deep learning?
It preserves variance in forward and backward passes with ReLU, reducing vanishing and exploding gradients. This stabilizes training and speeds convergence in deep networks.
Category: Relevancy
How does He Initialization differ from Xavier Initialization?
He uses variance 2/n for ReLU activations, while Xavier uses 1/n for tanh/sigmoid. He better suits rectifiers by accounting for their half-activation behavior.
Category: Comparison
When should He Initialization be applied?
Use it in layers with ReLU or variants like Leaky ReLU. Avoid using it for saturating nonlinearities where Xavier or LeCun may perform better.
Category: Usage
Does He Initialization work with batch normalization?
Yes, it complements batch normalization. Batch normalization stabilizes activations, while He Initialization ensures initial variance is appropriate.
Category: Integration
Is He Initialization available in popular frameworks?
Yes, both PyTorch and TensorFlow provide He/Kaiming normal and uniform initializers for easy application.
Category: Tools
What is the mathematical basis of He Initialization?
It sets weight variance proportional to 2 divided by the number of input units, reflecting ReLU’s zeroing of negative inputs.
Category: Theory
Can He Initialization be used for convolutional layers?
Yes, frameworks compute fan-in using kernel size and channel counts and apply He formulas accordingly.
Category: Usage
What common mistakes occur with He Initialization?
Using He with non-ReLU activations or miscomputing fan-in/fan-out can destabilize training. Mixing incompatible initializers across layers also causes issues.
Category: Mistakes
What are alternatives to He Initialization?
Xavier, LeCun, and orthogonal initialization are common alternatives, chosen based on activation function and architecture.
Category: Alternatives