DataTerminology.comNEW
Also known as: Glorot Initialization
A weight initialization strategy balancing variance across layers to stabilize gradients and improve neural network training.
DESCRIPTION
Xavier (Glorot) Initialization sets weights so that the variance of activations is approximately preserved across layers, mitigating vanishing or exploding gradients. It chooses bounds based on fan-in and fan-out, using uniform or normal distributions. Widely available in frameworks like PyTorch and Keras, it suits tanh and sigmoid activations. For ReLU-based networks, He initialization is often preferred. Proper initialization complements normalization and careful optimizer selection.
ALTERNATIVE DEFINITIONS
PMLR
We propose to initialize the weights so that the variance of the activations is the same across every layer.
States the core variance-preserving principle of the method.
PyTorch
Fills the input Tensor with values drawn from the uniform distribution U(−a, a) where a = gain * sqrt(6 / (fan_in + fan_out)).
Provides the practical formula used in the uniform variant.
Keras
Glorot uniform initializer draws samples from a uniform distribution within [-limit, limit], where limit = sqrt(6 / (fan_in + fan_out)).
Explicit initializer definition within Keras API.
RELATED TERMS
Weight Initialization Vanishing Gradient Activation Function Batch Normalization Deep Learning Gradient Descent Neural Network Variance Scaling Keras Initializers
COMPETING TERMS
He Initialization LeCun Initialization
RELATIONS
- Broader: Initialization
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - Artificial neural network
- PyTorch - Xavier uniform initializer doc
- Keras - Initializer API reference
- PMLR - Original method paper
TOP QUESTIONS
When should Xavier Initialization be used?
Use it with tanh or sigmoid activations to balance gradient flow across layers. For ReLU variants, prefer He initialization.
Category: Requirements
What problem does Xavier Initialization address?
It mitigates vanishing and exploding gradients by preserving activation variance across layers at initialization.
Category: Relevancy
What are common mistakes with Xavier Initialization?
Using it with ReLU without adjusting gain, or ignoring fan-in/fan-out values can destabilize training.
Category: Mistakes
What risks arise from poor initialization?
Models may fail to converge or require excessive epochs; gradients become unstable, harming performance and generalization.
Category: Risks
How does Xavier compare to He Initialization?
Xavier targets symmetric activations; He scales variance for ReLU, often improving deeper networks.
Category: Alternatives
How is Xavier Initialization implemented in practice?
Use framework functions (e.g., PyTorch or Keras) and set gain based on activation choice; confirm tensor shapes and reproducibility seeds.
Category: Management
Is Xavier Initialization still relevant with normalization layers?
Yes, good initialization complements batch or layer normalization, improving early training stability and speed.
Category: Relevancy
What is a striking analogy for Xavier Initialization?
Like balancing water pressure across connected pipes so flow stays even throughout the network.
Category: Example
Where did Xavier Initialization originate?
It was proposed by Glorot and Bengio in their 2010 paper on training deep feedforward networks.
Category: Origin