DataTerminology.comNEW
Also known as: Gaussian Error Linear Unit
Gaussian Error Linear Unit, a smooth activation used in transformers.
DESCRIPTION
GELU is a smooth nonlinearity that weights inputs by their value using the Gaussian cumulative distribution. It often improves performance over ReLU in transformer models like BERT. GELU combines benefits of saturation and linear regimes with better gradient properties. It can be implemented exactly or with fast approximations. Widely available in deep learning frameworks.
ALTERNATIVE DEFINITIONS
arXiv
We propose the Gaussian Error Linear Unit (GELU), which weights inputs by their value, rather than gates them by their sign as in ReLUs.
Original paper contrasting GELU with ReLU gating by sign.
PyTorch
Applies the Gaussian Error Linear Units (GELU) function: GELU(x) = x * Φ(x), where Φ(x) is the Cumulative Distribution Function of the standard normal distribution.
Gives exact functional form integrating the Gaussian CDF.
TensorFlow
Computes the Gaussian Error Linear Unit (GELU) activation function, a smooth nonlinearity that weights inputs by their value via the Gaussian CDF.
Framework-level definition highlighting smoothness and Gaussian CDF weighting.
RELATED TERMS
Sigmoid Activation Function Transformer Model BERT Nonlinearity Neural Network
COMPETING TERMS
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist Software Developer
EXTERNAL SOURCES
- Wikipedia - Activation function
- arXiv - Original GELU paper
- PyTorch - GELU activation in PyTorch
- TensorFlow - TensorFlow GELU op
TOP QUESTIONS
What is GELU?
GELU is a smooth activation that weights inputs by the Gaussian CDF, commonly used in transformers.
Category: Definition
Why prefer GELU over ReLU?
GELU often improves accuracy in transformer models and avoids hard zeroing of negatives, aiding gradient flow.
Category: Comparison
Is GELU computationally expensive?
Slightly more than ReLU, but frameworks provide efficient implementations and approximations.
Category: Requirements
Does GELU always outperform ReLU?
No. Benefits depend on architecture and task. Benchmark both where possible.
Category: Opinion
How is GELU implemented?
Either via exact Gaussian CDF or approximations like tanh-based formulas for speed.
Category: How-to
Where is GELU used?
Widely in transformer encoders like BERT and many modern language and vision models.
Category: Application
What are alternatives to GELU?
ReLU, Swish/SiLU, ELU, and Mish are common alternatives with different trade-offs.
Category: Alternatives
Any stability concerns with GELU?
Generally stable; ensure correct dtype and initialization to avoid saturation in extreme ranges.
Category: Risk
Does GELU require special initialization?
Standard initializations typically suffice; follow architecture-specific recommendations.
Category: BestPractice