DataTerminology.comNEW
An adaptive optimizer that combines momentum with per-parameter learning rates using estimates of first and second moments of gradients.
DESCRIPTION
Adam is a first-order optimizer that maintains exponential moving averages of gradients and their squared values, providing adaptive step sizes for each parameter. Bias-correction ensures unbiased estimates early in training. It works well for sparse gradients and is a strong default for many deep learning tasks. However, generalization can lag behind tuned SGD in some vision tasks. Variants like AdamW decouple weight decay for better regularization behavior.
ALTERNATIVE DEFINITIONS
arXiv
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments.
Concise original definition emphasizing adaptive moment estimates.
TensorFlow
Optimizer that implements the Adam algorithm.
Implementation-focused API description for practical use.
PyTorch
Implements Adam algorithm.
Minimal API definition indicating supported optimizer behavior.
RELATED TERMS
Learning Rate Schedule L2 Regularization Adaptive Optimizers Bias Correction Beta1 Beta2 Epsilon
COMPETING TERMS
SGD with Momentum RMSProp AdamW
RELATIONS
- Broader: Adaptive Optimizers Optimizer
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Adam (optimization algorithm)
- arXiv - Original Adam paper
- TensorFlow - Keras Adam optimizer API
- PyTorch - PyTorch Adam optimizer docs
- arXiv - AdamW decoupled weight decay
TOP QUESTIONS
When should I use the Adam Optimizer?
Adam is a strong default for many deep learning tasks, especially with sparse or noisy gradients. It converges quickly with minimal tuning.
Category: Usage
What are the key hyperparameters of Adam?
Learning rate, beta1, beta2, and epsilon. Defaults (0.001, 0.9, 0.999, 1e-7/1e-8) work well but should be tuned for specific tasks.
Category: Tuning
Why might SGD with momentum generalize better than Adam?
Adam’s adaptivity can lead to different implicit regularization. In some vision tasks, tuned SGD achieves better test performance despite slower convergence.
Category: Opinion
What is bias correction in Adam?
It corrects the moving average estimates for their initial bias toward zero, especially in early steps, improving stability.
Category: Concept
How does Adam relate to RMSProp and momentum?
Adam merges RMSProp-like second-moment adaptation with momentum on first moments, providing per-parameter adaptive updates.
Category: Differences
What are common mistakes when using Adam?
Leaving default learning rate too high, misusing weight decay (not using AdamW), or forgetting warmup in transformers can hurt results.
Category: Mistakes
Is Adam suitable for large-batch training?
Yes, but you may need learning-rate scaling and warmup. Validate convergence and generalization carefully.
Category: BestPractices
How does AdamW improve upon Adam?
AdamW decouples weight decay from the gradient step, aligning it with true L2 regularization behavior and often improving generalization.
Category: Differences
Should I use learning rate schedules with Adam?
Often yes. Warmup and cosine decay are common and can stabilize and improve convergence.
Category: Tuning