DataTerminology.comNEW
Also known as: LR Schedule
A predefined policy for varying the learning rate during training to improve convergence speed, stability, and generalization.
DESCRIPTION
Learning rate schedules modulate the step size over time to balance fast learning with stability. Common strategies include step decay, exponential decay, cosine annealing with warm restarts, cyclical schedules, and OneCycle policies. Warmup helps stabilize early training, especially in deep transformers. Schedules interact with optimizer choice and batch size and can significantly impact final accuracy. Proper scheduling is a key lever for training efficiency.
ALTERNATIVE DEFINITIONS
TensorFlow
Learning rate schedules provide a way to vary the learning rate over time during training.
Concise API-level definition stressing time-varying learning rates.
PyTorch
Learning rate scheduling is a way of decreasing the learning rate during training according to some policy.
Emphasizes policy-driven decreases during training.
arXiv
We propose SGDR: stochastic gradient descent with warm restarts, where the learning rate follows a cosine annealing schedule and is periodically increased to the initial value.
Introduces cosine annealing with warm restarts as a practical schedule.
RELATED TERMS
Adam Optimizer Stochastic Gradient Descent Warmup Cosine Annealing One-Cycle Policy Exponential Decay Step Decay Cyclical Learning Rates Early Stopping Superclass
COMPETING TERMS
Constant Learning Rate Adaptive Optimizers
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist MLOps Engineer AI Engineer
EXTERNAL SOURCES
- Wikipedia - Learning rate
- TensorFlow - Keras schedules API docs
- PyTorch - PyTorch LR scheduling guide
- arXiv - SGDR cosine annealing schedule
- arXiv - Cyclical learning rates paper
TOP QUESTIONS
Why use a Learning Rate Schedule instead of a constant rate?
Schedules accelerate early learning and stabilize convergence later. They often yield higher final accuracy and better generalization.
Category: Relevancy
How does warmup help Learning Rate Schedules?
Warmup gradually increases the rate at the start, preventing instability with large initial steps, especially in deep or transformer models.
Category: Concept
What schedule should I start with?
Cosine decay with warmup is a strong default. OneCycle and step decay are also effective; validate empirically.
Category: BestPractices
Do Learning Rate Schedules work with adaptive optimizers like Adam?
Yes. Schedules and adaptive updates are complementary and often improve convergence together.
Category: Usage
What are common mistakes with Learning Rate Schedules?
Too aggressive decay can stall learning; too mild decay can overfit. Forgetting to align schedule steps to epoch/iteration counts causes mismatch.
Category: Mistakes
How do batch size and schedule interact?
Larger batches often need higher initial rates and longer warmup. Scale schedule parameters accordingly.
Category: Tuning
What metrics indicate a schedule is working?
Smoothly decreasing loss, stable gradients, and improved validation accuracy. Learning curves without oscillations suggest appropriate rates.
Category: Evaluation
Are restarts beneficial in Learning Rate Schedules?
Warm restarts can help escape local minima and improve exploration. SGDR is a common approach.
Category: Opinion
Can Learning Rate Schedules reduce training time?
Often yes, by enabling larger effective steps early and converging faster. They also reduce the need for extensive manual retuning.
Category: Benefits