DataTerminology.comNEW
The hyperbolic tangent activation function that maps real numbers to the range −1 to 1, often used to center neuron outputs.
DESCRIPTION
Tanh is a smooth, S-shaped activation function frequently used in neural networks to provide zero-centered outputs. It is a rescaled version of the logistic sigmoid and can improve optimization by centering activations. However, tanh saturates for large inputs, potentially causing vanishing gradients in deep networks. Modern architectures often prefer ReLU variants, but tanh remains useful in recurrent networks and certain normalization contexts.
ALTERNATIVE DEFINITIONS
Wikipedia
The hyperbolic tangent is a mathematical function, tanh x, which is defined as the ratio of the hyperbolic sine and the hyperbolic cosine.
Provides the precise mathematical definition via hyperbolic functions.
Deep Learning (Goodfellow, Bengio, Courville)
The tanh function squashes real numbers to the range (−1, 1) and is a rescaled, shifted version of the logistic sigmoid.
Explains relationship to sigmoid and output range in deep learning context.
PyTorch
Applies the Tanh function element-wise.
Operational definition as used in neural network layers.
RELATED TERMS
Activation Function Backpropagation Vanishing Gradient Neural Network Softsign Swish Batch Normalization Gradient Descent
COMPETING TERMS
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Hyperbolic tangent
- PyTorch - API docs for Tanh layer
- TensorFlow - API docs for tf.nn.tanh
- Deep Learning Book - Theory of activations including tanh
TOP QUESTIONS
When should I use Tanh instead of Sigmoid?
Use tanh when zero-centered activations help optimization, especially in shallow networks or RNNs. Sigmoid outputs are non-centered and can slow convergence.
Category: Comparison
Why does Tanh cause vanishing gradients?
For large positive or negative inputs, tanh saturates near ±1, making derivatives near zero. This reduces gradient flow through deep layers.
Category: Concept
How can I mitigate Tanh saturation?
Use careful initialization, batch normalization, or residual connections. Alternatively, consider ReLU-family activations that do not saturate in the positive region.
Category: BestPractice
Is Tanh still relevant in modern networks?
Yes, particularly in gated RNNs like LSTMs and GRUs, and when centered activations are helpful. ReLU variants are more common for deep feed-forward nets.
Category: Relevancy
Does Tanh require specific weight initialization?
Glorot (Xavier) initialization works well for tanh. Proper scaling helps maintain signal variance through layers.
Category: Requirements
What’s a risk of choosing Tanh in deep models?
Excessive depth with tanh can impede training due to vanishing gradients. Model may underfit or converge slowly.
Category: Risks
How does Tanh compare with ReLU in practice?
ReLU is simple and avoids saturation for positive inputs, often training faster. Tanh may yield better centered activations in some cases.
Category: Opinion
Is Tanh differentiable everywhere?
Yes, tanh is smooth and differentiable for all real inputs, aiding gradient-based optimization.
Category: Concept
What is the output range of Tanh?
Tanh outputs lie strictly between −1 and 1, centering around zero.
Category: Basics