DataTerminology.comNEW
Also known as: Decoupled Weight Decay, Adam with Weight Decay
An Adam variant that decouples weight decay from gradient-based updates to improve generalization.
DESCRIPTION
AdamW modifies Adam by decoupling weight decay from the gradient update, aligning the behavior of weight decay with its original regularization intent. This improves convergence and generalization, especially for deep networks such as transformers. AdamW is widely used in NLP and CV, often combined with learning rate schedules (e.g., cosine decay) and warmup. Its update rule applies L2-style decay directly to parameters rather than conflating it with gradients. This decoupling enables clearer hyperparameter tuning and more stable training. Frameworks like PyTorch, TensorFlow/Keras, and JAX include AdamW as a first-class optimizer.
ALTERNATIVE DEFINITIONS
Loshchilov & Hutter (ICLR 2019)
We propose decoupling weight decay from the gradient-based update of Adam to achieve better training and generalization performance.
Highlights the key idea: decoupling weight decay from gradients to fix Adam’s regularization.
PyTorch
Implements AdamW algorithm.
Concise implementation-focused description from a major deep learning framework.
RELATED TERMS
L2 Regularization Learning Rate Cosine Annealing Warmup Optimizer Transformer Model
COMPETING TERMS
Adam SGD with Momentum RMSProp
RELATIONS
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Adam (optimization algorithm)
- arXiv - Original AdamW paper
- Keras - Keras AdamW optimizer docs
TOP QUESTIONS
Why use AdamW instead of Adam?
AdamW decouples weight decay from gradients, which typically yields better generalization and more predictable regularization behavior than Adam’s L2 via gradient modification.
Category: Relevancy
How should weight decay be tuned for AdamW?
Start with small values (e.g., 1e-4 to 1e-2) and adjust based on validation loss and overfitting signs. Tune jointly with learning rate and schedule.
Category: Requirements
What models benefit most from AdamW?
Transformer-based models in NLP and vision often benefit, as AdamW stabilizes training and improves generalization compared to Adam.
Category: Example
Is AdamW compatible with learning rate warmup and cosine decay?
Yes. It’s commonly paired with warmup and cosine decay to improve early stability and long-run convergence.
Category: BestPractice
What is a common mistake when using AdamW?
Applying weight decay to parameters that should be excluded, such as bias terms and LayerNorm/BatchNorm parameters, can degrade performance.
Category: Mistakes
How does AdamW affect training stability?
By separating regularization from gradients, AdamW avoids unintended interactions that can destabilize updates, improving stability.
Category: Benefits
When might SGD with momentum outperform AdamW?
In some vision tasks with extensive training and data augmentation, tuned SGD with momentum can match or exceed AdamW generalization.
Category: Opinion
Does AdamW increase computational cost?
No significant additional cost versus Adam; it mainly changes how weight decay is applied in updates.
Category: Costs
How does AdamW interact with gradient clipping?
Use clipping on gradients before applying decoupled weight decay; they control different aspects of training stability.
Category: BestPractice