DataTerminology.comNEW
A small subset of training examples processed together in one optimization step during learning.
DESCRIPTION
Minibatching partitions training data into small subsets so models update parameters more frequently than full-batch training while retaining vectorization efficiency. Typical sizes range from tens to thousands of examples, constrained by memory and model architecture. Minibatches smooth gradient estimates, improve hardware utilization, and enable stable, scalable training. Choosing batch size involves trade-offs among generalization, convergence speed, and throughput. Modern frameworks and accelerators are optimized for minibatch processing across devices.
ALTERNATIVE DEFINITIONS
Google Developers
mini-batch: A small subset of the training set selected and used to train a model during each iteration.
Emphasizes subset selection per training iteration for updates.
DeepLearning.AI
A mini-batch is a small, random subset of the training data used to compute an estimate of the gradient and update model parameters.
Highlights gradient estimation and parameter update role.
RELATED TERMS
Batch Size Epoch Learning Rate DataLoader Convergence Superclass Batch Normalization Optimizer
COMPETING TERMS
Stochastic Gradient Descent Batch Gradient Descent
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer Data Engineer
EXTERNAL SOURCES
- Wikipedia - Stochastic gradient descent
- Stanford CS231n - Mini-batch gradient descent notes
- PyTorch - DataLoader batching utilities
- TensorFlow - Efficient input pipelines
TOP QUESTIONS
Why use a Minibatch instead of full-batch training?
Minibatches reduce memory footprint and provide more frequent updates, accelerating convergence. They leverage parallel hardware efficiently while offering smoother gradients than pure stochastic updates.
Category: Relevancy
How do Minibatch sizes affect generalization?
Smaller batches introduce gradient noise that can help escape sharp minima and improve generalization. Very large batches may converge faster but can overfit or generalize worse without tuning.
Category: Opinion
What is a common mistake when choosing Minibatch size?
Selecting based solely on throughput without monitoring validation metrics. Batch size should be co-tuned with learning rate, regularization, and optimizer for stable training.
Category: Mistakes
How does Minibatch relate to epochs and steps?
An epoch is one pass over the dataset. Steps are minibatch updates; steps per epoch equal dataset size divided by batch size.
Category: Example
What optimizer considerations apply to Minibatch training?
Adaptive optimizers and learning rate schedules can stabilize training. Linear scaling of learning rate with batch size is a useful heuristic for large batches.
Category: Requirements
How does distributed training change Minibatch handling?
Global batch size equals local batch size times number of workers. Gradient synchronization and learning rate scaling must be carefully managed.
Category: Management
Are Minibatches always randomly sampled?
Random shuffling each epoch is standard to reduce bias. However, curriculum learning or class-balanced sampling may improve convergence for specific tasks.
Category: Topic
What risks arise with very small Minibatches?
Excessive gradient noise can destabilize training and slow convergence. Batch normalization may underperform with tiny batches unless adapted (e.g., group or layer norm).
Category: Risks
How do Minibatches interact with Batch Normalization?
BatchNorm relies on batch statistics; small batches can yield noisy estimates. Alternatives like GroupNorm or using accumulated statistics can mitigate issues.
Category: Misunderstanding