DataTerminology.comNEW
Also known as: Momentum SGD
Stochastic gradient descent augmented with a velocity term to accelerate convergence and damp oscillations.
DESCRIPTION
SGD with momentum accumulates a moving average of past gradients to continue updates in consistent directions and reduce zig-zagging. The momentum term acts like inertia, improving convergence speed on noisy or ill-conditioned surfaces. It is simple to implement and often outperforms plain SGD on deep networks. Tuning learning rate and momentum coefficient is critical for stability. Variants include Nesterov momentum, which looks ahead before applying gradients.
ALTERNATIVE DEFINITIONS
Wikipedia
Stochastic gradient descent (SGD) is an iterative method for optimizing an objective function with suitable smoothness properties.
Defines SGD broadly; momentum is a known extension to accelerate convergence.
PyTorch
Implements stochastic gradient descent (optionally with momentum).
Concise API-level description highlighting optional momentum.
Keras
SGD optimizer implements the stochastic gradient descent algorithm with support for momentum, learning rate decay, and Nesterov momentum.
Notes support for momentum and Nesterov within a practical optimizer.
RELATED TERMS
Stochastic Gradient Descent Gradient Descent Learning Rate Loss Function Backpropagation Optimization Convergence
COMPETING TERMS
Adam Nesterov Momentum RMSProp
RELATIONS
- Broader: Momentum
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Stochastic gradient descent
- PyTorch - SGD optimizer API with momentum
- Keras - Keras SGD optimizer doc
- Stanford CS231n - Optimization primer including momentum
TOP QUESTIONS
Why use SGD with Momentum instead of plain SGD?
Momentum accelerates learning along consistent gradient directions and reduces oscillations, improving convergence speed on noisy or curved loss landscapes.
Category: Benefits
How do you tune the momentum coefficient in SGD with Momentum?
Typical values range 0.8–0.99. Start at 0.9, adjust based on training stability, validation performance, and learning rate interactions.
Category: HowTo
What is the difference between Momentum and Nesterov Momentum?
Nesterov evaluates gradients at the look-ahead position, often yielding better convergence and stability than classical momentum.
Category: Concepts
How does SGD with Momentum compare to Adam?
Adam adapts per-parameter learning rates and includes momentum-like terms; it can converge faster but may generalize worse in some settings.
Category: Alternatives
When is SGD with Momentum preferred?
It’s often favored for large-scale vision models and when generalization is key. Simplicity and predictable behavior aid reproducibility.
Category: Opinion
What common mistakes occur using SGD with Momentum?
Too high learning rate with high momentum can diverge. Forgetting to decay learning rate can stall convergence.
Category: Mistakes
Does batch size affect SGD with Momentum?
Yes. Larger batches reduce gradient noise, potentially allowing higher momentum and learning rates; small batches may need conservative settings.
Category: Requirements
Can momentum help escape local minima?
Momentum can roll through shallow minima and plateaus by carrying update inertia, aiding exploration of better basins.
Category: Theory
Is weight decay compatible with SGD with Momentum?
Yes. Many frameworks combine momentum and weight decay (L2) to improve generalization and stability.
Category: Concepts