DataTerminology.comNEW
Also known as: Adaptive Moment Estimation
An adaptive optimizer combining momentum and RMSProp ideas to adjust learning rates per parameter.
DESCRIPTION
Adam is a first-order stochastic optimizer that maintains exponential moving averages of gradients and squared gradients to adapt step sizes. It typically converges faster than vanilla SGD and is widely used as a default for deep networks. While effective, Adam may generalize worse than tuned SGD in some regimes. Variants like AMSGrad address theoretical convergence, and AdamW decouples weight decay for better regularization. Hyperparameters β1, β2, and ε control momentum and numerical stability.
ALTERNATIVE DEFINITIONS
arXiv
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.
Original paper highlighting adaptive moment estimates and stochastic objectives.
Wikipedia
Adam is an adaptive learning rate optimization algorithm designed specifically for training deep neural networks.
Concise description emphasizing adaptive learning rates for deep learning.
Keras
Optimizer that implements the Adam algorithm.
API-level definition used in practice with default hyperparameters.
PyTorch
Implements Adam algorithm.
Emphasizes the implementation in a widely used deep learning framework.
RELATED TERMS
Stochastic Gradient Descent Momentum AMSGrad AdamW Learning Rate Scheduler L2 Regularization Gradient Clipping Optimization Algorithm
COMPETING TERMS
SGD with Momentum RMSProp Adagrad
RELATIONS
- Broader: Optimization Algorithm
- Derived from: Stochastic Gradient Descent
- Narrower: AdamW
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Adam (optimization algorithm)
- arXiv - Original Adam paper by Kingma and Ba
- Keras - Keras optimizer documentation
- PyTorch - PyTorch optimizers overview
TOP QUESTIONS
When should I choose Adam over SGD with momentum?
Use Adam for fast convergence and uneven gradient scales, especially early in training or with sparse features. For best generalization on vision tasks, tuned SGD often wins.
Category: Opinion
What do β1 and β2 control in Adam?
β1 controls momentum on first moments (gradients) and β2 controls momentum on second moments (squared gradients). Typical defaults are β1=0.9 and β2=0.999.
Category: Concepts
What is the difference between Adam and AdamW?
AdamW decouples weight decay from the gradient update, improving regularization fidelity and often generalization compared to Adam’s L2-regularization-like behavior.
Category: Comparison
Why can Adam sometimes generalize worse than SGD?
Adam’s adaptivity can converge to sharper minima. With careful learning rate schedules and momentum tuning, SGD may find flatter, better-generalizing solutions.
Category: Misunderstanding
How sensitive is Adam to the learning rate?
Adam is more forgiving than SGD but still benefits from tuning. Start around 1e-3 and adjust with warmup or cosine decay for stability and performance.
Category: Practice
What is AMSGrad and why was it proposed?
AMSGrad modifies Adam’s second-moment estimate to enforce non-increasing step sizes, addressing certain convergence pathologies in the original algorithm.
Category: Concepts
Does Adam work well for reinforcement learning?
Yes, Adam is common in RL due to noisy gradients, but careful tuning, clipping, and schedules are crucial for stability.
Category: UseCase
Should I use gradient clipping with Adam?
Yes. Clipping helps tame exploding updates in unstable regimes, improving training stability across tasks.
Category: BestPractice
What are common mistakes when using Adam?
Leaving defaults when loss plateaus, forgetting weight decay decoupling (AdamW), and not scheduling the learning rate are frequent pitfalls.
Category: Mistakes