DataTerminology.comNEW
An S-shaped function mapping real-valued inputs to a bounded range, often used to model probabilities.
DESCRIPTION
The sigmoid (logistic) function σ(x)=1/(1+e^(−x)) maps inputs to (0,1), making it a natural choice for probability outputs in binary classification and logistic regression. In neural networks, sigmoid activations historically dominated but can cause vanishing gradients in deep layers. Alternatives like ReLU and tanh often perform better for hidden layers, while sigmoid remains standard for output layers in binary tasks. Proper initialization and normalization mitigate numerical saturation.
ALTERNATIVE DEFINITIONS
Wikipedia
A sigmoid function is a bounded, differentiable, real function that is defined for all real input values and has a non-negative derivative at each point.
Highlights boundedness, differentiability, and monotonicity.
Wolfram MathWorld
The logistic function is given by f(x) = 1/(1+e^{-x}).
Provides the explicit functional form of the logistic sigmoid.
RELATED TERMS
Logistic Regression Activation Function Softmax Vanishing Gradient Cross-Entropy Loss Relative Frequency Binary Classification Neural Network
COMPETING TERMS
KNOWLEDGE AREAS
TARGET AUDIENCE
EXTERNAL SOURCES
- Wikipedia - Sigmoid function
- Deep Learning (Goodfellow, Bengio, Courville) - Activation functions and multilayer perceptrons
- Stanford CS231n - Practical notes on activation functions
TOP QUESTIONS
When should I use Sigmoid in neural networks?
Use sigmoid for binary classification outputs to represent probabilities. Avoid in deep hidden layers due to saturation and vanishing gradients; consider ReLU/tanh instead.
Category: Application
Why does Sigmoid cause vanishing gradients?
Near its asymptotes, the derivative approaches zero, shrinking gradient signals through layers and slowing or stalling learning.
Category: Concept
How does Sigmoid differ from Tanh?
Sigmoid outputs in (0,1), while tanh outputs in (−1,1). Tanh is zero-centered, often easing optimization in hidden layers.
Category: Alternatives
What is the logistic regression connection to Sigmoid?
Logistic regression models log-odds linearly and applies the sigmoid to produce probabilities for binary outcomes.
Category: Basics
What are common mistakes with Sigmoid outputs?
Using mean squared error instead of cross-entropy can hinder learning. Also, not clipping logits can cause numerical instability.
Category: Mistakes
How to mitigate Sigmoid saturation?
Use proper weight initialization, batch normalization, or prefer alternative activations in hidden layers. Calibrate learning rates carefully.
Category: BestPractices
Is Sigmoid differentiable everywhere?
Yes, sigmoid is smooth and differentiable for all real inputs, facilitating gradient-based optimization.
Category: Theory
What loss pairs naturally with Sigmoid?
Binary cross-entropy (log loss) is the canonical loss for sigmoid outputs in binary classification.
Category: Requirements
How do I interpret Sigmoid output 0.8?
It suggests an 80% model-estimated probability of the positive class, assuming well-calibrated outputs.
Category: Interpretation