DataTerminology.comNEW
Also known as: Adaptive Moment Estimation
A first-order optimizer that adapts learning rates using estimates of first and second moments of gradients.
DESCRIPTION
Adam is an adaptive gradient-based optimizer widely used to train deep neural networks. It maintains exponentially decaying averages of past gradients and squared gradients to adjust learning rates per parameter. Default hyperparameters often work well, making it a popular baseline. Variants like AdamW decouple weight decay from the gradient update to improve generalization. While robust, Adam can converge to suboptimal minima or show poor generalization without tuning. Monitoring learning dynamics and regularization is recommended.
ALTERNATIVE DEFINITIONS
Kingma & Ba (Adam paper)
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.
Original paper emphasizes moment estimates and stochastic objectives.
PyTorch
Implements Adam algorithm.
Concise implementation reference for practitioners.
RELATED TERMS
Stochastic Gradient Descent Momentum Learning Rate Scheduling Optimizer Gradient Descent
COMPETING TERMS
SGD with Momentum RMSProp Adagrad
RELATIONS
- Broader: Optimization Optimizer
- Derived from: Stochastic Gradient Descent
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Stochastic gradient descent#Adam
- arXiv - Original Adam optimizer paper
- PyTorch - Adam class API and defaults
- TensorFlow - Keras Adam optimizer details
TOP QUESTIONS
Why is Adam Optimization so popular?
It adapts learning rates per parameter and typically works well with minimal tuning, accelerating training and improving stability.
Category: Relevancy
When should I prefer SGD with Momentum over Adam Optimization?
SGD with momentum can yield better generalization in some vision tasks; try Adam first, then compare validation performance.
Category: Opinion
What are Adam’s default hyperparameters?
Common defaults are lr=0.001, beta1=0.9, beta2=0.999, eps=1e-8; these are good starting points.
Category: Requirements
What is AdamW and why use it?
AdamW decouples weight decay from the gradient update, often improving generalization and tuning clarity.
Category: Alternatives
What pitfalls exist with Adam Optimization?
It may converge to poorer minima or overfit; consider learning rate schedules, weight decay, and validation monitoring.
Category: Risks
How does Adam Optimization work internally?
It computes exponential moving averages of gradients and squared gradients, correcting bias and adjusting step sizes per parameter.
Category: Example
Is Adam suitable for large language models?
Yes; Adam and AdamW are standard for transformer training with appropriate scaling and schedules.
Category: Use cases
How do I tune Adam’s learning rate?
Use learning rate ranges, warmup, and cosine decay. Start near 1e-3 for small nets and lower for large models.
Category: Best practices
Does Adam guarantee convergence?
No; though effective in practice, convergence depends on problem conditions and hyperparameter choices.
Category: Misunderstanding