DataTerminology.comNEW
Also known as: SGD Momentum, Classical Momentum
An optimization technique that accumulates past gradients to accelerate convergence and reduce oscillations in parameter updates.
DESCRIPTION
Momentum adds a fraction of the previous update to the current gradient step, building velocity toward minima while damping oscillations. It helps traverse ravines and improves convergence speed, especially with noisy gradients. The momentum coefficient (e.g., 0.9) controls the influence of past updates. Nesterov momentum anticipates the future position for more stable improvements. Momentum is widely used in training deep neural networks and pairs well with learning rate schedules.
ALTERNATIVE DEFINITIONS
Wikipedia
Momentum accelerates gradient descent by taking into account the previous gradients to smooth out the updates.
Highlights smoothing effect by incorporating previous gradients.
Stanford CS231n
The idea of Momentum is to build up velocity by adding a fraction of the previous update vector to the current update.
Emphasizes velocity intuition and the additive update rule.
RELATED TERMS
Gradient Descent Learning Rate Optimization Loss Function Stochastic Gradient Descent RMSProp Convergence Batch Size
COMPETING TERMS
RELATIONS
- Narrower: Nesterov Momentum SGD with Momentum
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Stochastic gradient descent
- TensorFlow - SGD optimizer with momentum
- PyTorch - PyTorch SGD with momentum docs
TOP QUESTIONS
What is Momentum in optimization?
Momentum accumulates past updates to accelerate gradient descent and reduce oscillations, improving convergence speed in deep learning training.
Category: Definition
How does Momentum help training?
It builds velocity in consistent directions and dampens oscillations across steep dimensions of the loss surface, speeding up progress.
Category: Example
What value should be used for Momentum?
Typical values are 0.9 or 0.95, but tuning is dataset and architecture dependent and should be validated.
Category: Requirements
What is Nesterov Momentum?
It computes the gradient at the anticipated future position, often yielding more stable and effective updates.
Category: Definition
What are the risks of using Momentum?
Excessive momentum can overshoot minima or cause divergence; combine with appropriate learning rates and schedules.
Category: Risks
How does Momentum compare to Adam?
Adam adds adaptive learning rates to momentum-like updates, often converging faster but sometimes generalizing worse.
Category: Alternatives
Is Momentum useful with small batches?
Yes. It helps smooth noisy gradients typical of small-batch training.
Category: Opinion
How to monitor Momentum’s effect?
Track loss trajectories and update norms; look for smoother decreases and fewer oscillations.
Category: Testing
Where to learn more about Momentum?
Study CS231n notes and optimizer API docs in TensorFlow or PyTorch.
Category: Overview