DataTerminology.comNEW
Also known as: Rectified Linear Unit
Piecewise linear activation defined as f(x) = max(0, x), promoting sparse activations and faster training.
DESCRIPTION
The Rectified Linear Unit (ReLU) outputs the input for positive values and zero otherwise. Its non-saturating gradient mitigates vanishing gradients, accelerating deep network training. ReLU encourages sparsity, aiding generalization and computational efficiency. Variants like Leaky ReLU, Parametric ReLU, and GELU address dead neuron and smoothness issues. Despite simplicity, careful initialization and normalization are needed to avoid exploding activations.
ALTERNATIVE DEFINITIONS
Wikipedia
The rectified linear unit (ReLU) is a piecewise linear function that will output the input directly if it is positive, otherwise, it will output zero.
Defines behavior succinctly, emphasizing piecewise linearity.
Stanford CS231n
ReLU: f(x) = max(0, x).
Gives the exact mathematical definition used in practice.
Goodfellow, Bengio, Courville
The rectified linear activation function is f(z)=max(0,z).
Presents ReLU within the broader theory of deep networks.
RELATED TERMS
Activation Function Gradient Vanishing Backpropagation Neural Network Convolutional Neural Network Batch Normalization Xavier Initialization
COMPETING TERMS
Logistic Function Tanh Function Leaky ReLU
RELATIONS
- Narrower: Leaky ReLU
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Rectifier (neural networks)
- Stanford CS231n - Activation functions overview
- Keras - ReLU layer API
- PyTorch - ReLU module docs
TOP QUESTIONS
Why is ReLU widely used in deep networks?
ReLU mitigates vanishing gradients and is computationally cheap. It promotes sparse activations, often improving generalization and training speed.
Category: Relevancy
What is the main downside of ReLU?
Neurons can die (output zero permanently) when gradients push weights to always-negative inputs. Proper initialization and learning rates help mitigate this.
Category: Risks
When should I use Leaky ReLU instead of ReLU?
Use Leaky ReLU to reduce dead neuron risk, especially in very deep or poorly initialized networks, by allowing a small negative slope.
Category: Opinion
Does ReLU work well for all tasks?
It works broadly but may be suboptimal for some distributions. Alternatives like GELU or Swish can improve performance in certain transformer architectures.
Category: Opinion
How does ReLU affect gradient flow?
For positive inputs, the gradient is 1, preserving signal. For negative inputs, gradient is 0, truncating flow and potentially causing dead units.
Category: Example
Is ReLU differentiable?
ReLU is not differentiable at zero but is subdifferentiable. In practice, autodiff frameworks handle this with subgradient conventions.
Category: Misunderstanding
How do I initialize weights when using ReLU?
He (Kaiming) initialization is recommended to maintain variance through layers with ReLU activations.
Category: Best practice
Where should I place Batch Normalization relative to ReLU?
Common practice is Conv/Linear → BatchNorm → ReLU. Empirically, this ordering stabilizes training in many CNNs.
Category: Best practice
What is a good analogy for ReLU?
ReLU is like a one-way valve: it passes positive pressure (signals) while blocking negative flow.
Category: Example