DataTerminology.comNEW
Optimization method updating parameters using gradients computed on small, randomly sampled batches.
DESCRIPTION
Mini-batch gradient descent computes gradients on small subsets of training data to update model parameters. It balances the stability of batch gradient descent with the efficiency and generalization benefits of stochastic updates. Mini-batches leverage parallel hardware like GPUs and TPUs for throughput. Common batch sizes range from 16 to 4096, chosen based on model, data, and hardware. Proper shuffling, learning rate schedules, and regularization are critical for convergence. Optimizers like Adam or momentum variants are often used with mini-batches to accelerate training. In production, batch sizing also affects memory, latency, and reproducibility.
ALTERNATIVE DEFINITIONS
Wikipedia
Mini-batch gradient descent computes the gradient of the loss function for a subset of the training data and updates the parameters accordingly.
Contrasts mini-batch with full-batch and per-sample (stochastic) variants.
Google ML Crash Course
Mini-batch gradient descent uses small batches of training examples to compute each update to the model’s parameters.
Emphasizes the use of small batches for each update step.
RELATED TERMS
Learning Rate Momentum Adam Loss Function Backpropagation Epoch
COMPETING TERMS
Batch Gradient Descent Stochastic Gradient Descent
KNOWLEDGE AREAS
AI Mathematics Data Science MLOps
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer Software Developer
EXTERNAL SOURCES
- Wikipedia - Stochastic gradient descent
- Stanford CS231n - Notes on gradient descent variants
- PyTorch - Optimizers used with mini-batches
- TensorFlow - Keras training with batches
TOP QUESTIONS
Why use Mini-batch Gradient Descent?
It offers faster training than full-batch and less noisy updates than pure SGD. It also exploits GPU parallelism efficiently.
Category: Relevancy
How do you choose mini-batch size?
Balance hardware memory, throughput, and generalization. Start with powers of two (e.g., 32–256) and tune empirically with learning rate adjustments.
Category: Requirements
How does batch size affect generalization?
Very large batches can harm generalization by converging to sharper minima. Smaller batches introduce beneficial noise but may slow convergence.
Category: Opinion
What optimizers pair well with mini-batches?
Momentum, Adam, RMSProp, and Adagrad commonly accelerate convergence and stabilize updates.
Category: Best practices
Should data be shuffled for Mini-batch Gradient Descent?
Yes. Shuffling reduces correlation and bias in batches, improving convergence. Use deterministic seeds for reproducibility when needed.
Category: Best practices
What are common mistakes with Mini-batch Gradient Descent?
Using too-large batches for memory, skipping shuffling, and not scaling learning rate with batch size are common issues.
Category: Mistakes
How do learning rate schedules interact with batch size?
Larger batches often require proportionally higher learning rates and warm-up schedules; smaller batches may benefit from decay.
Category: Requirements
How does mixed precision affect mini-batching?
Mixed precision reduces memory and increases throughput, enabling larger batches but requiring loss scaling to maintain stability.
Category: Application
How to monitor training with Mini-batch Gradient Descent?
Track training/validation loss, gradient norms, and throughput. Log batch-level metrics to diagnose divergence or bottlenecks.
Category: Management