DataTerminology.comNEW
A training technique that mixes lower-precision (e.g., FP16/BF16) and FP32 arithmetic to reduce memory usage and increase throughput while preserving accuracy.
DESCRIPTION
Mixed precision accelerates training using FP16 or BF16 for most operations while retaining FP32 for numerically sensitive steps. Loss scaling mitigates underflow in gradients. Modern GPUs’ Tensor Cores optimize lower-precision math for speed. Benefits include reduced VRAM consumption and higher throughput. It is widely supported by PyTorch AMP and TensorFlow. Careful validation ensures accuracy remains comparable to full precision.
ALTERNATIVE DEFINITIONS
PyTorch
Automatic Mixed Precision (AMP) enables the use of lower-precision (FP16, BF16) operations with autocast and gradient scaling to improve performance and reduce memory without sacrificing model accuracy.
Describes practical AMP components: autocast and GradScaler.
TensorFlow
Mixed precision uses 16-bit floating point types to speed up training on GPUs/TPUs and reduce memory usage, typically with minimal impact on model quality.
Highlights hardware accelerators and memory savings.
NVIDIA Developer Blog
Mixed precision training uses FP16 for most operations while maintaining FP32 for some to achieve faster training and lower memory consumption with comparable accuracy.
Explains FP16/FP32 split and benefits on NVIDIA GPUs.
RELATED TERMS
FP16 BF16 FP32 Automatic Mixed Precision Loss Scaling Tensor Cores VRAM Throughput Quantization Training Stability
COMPETING TERMS
Full Precision Training Quantization-Aware Training
RELATIONS
- Broader: Training
KNOWLEDGE AREAS
AI MLOps Infrastructure Data Science
TARGET AUDIENCE
AI Engineer MLOps Engineer Data Scientist Hardware Engineer
EXTERNAL SOURCES
- Wikipedia - Numerical analysis#Floating-point arithmetic
- PyTorch - AMP documentation and examples
- TensorFlow - Guide for mixed precision
- NVIDIA - Blog on mixed precision benefits
TOP QUESTIONS
Why is Mixed Precision Training popular?
It increases throughput and reduces memory while keeping accuracy near FP32, leveraging Tensor Cores in modern accelerators.
Category: Relevancy
What is loss scaling in Mixed Precision Training?
It multiplies the loss to avoid gradient underflow in low precision, then scales gradients back during the update.
Category: Example
Does Mixed Precision Training always match FP32 accuracy?
Typically yes, but sensitive models may need tuning or selective FP32 operations.
Category: Opinion
How does Mixed Precision Training reduce VRAM usage?
FP16/BF16 halves tensor sizes, allowing larger batches or models.
Category: Example
What are common mistakes with Mixed Precision Training?
Disabling gradient scaling or casting too aggressively can destabilize training.
Category: Mistakes
Which hardware benefits most from Mixed Precision Training?
GPUs with Tensor Cores (NVIDIA) and TPUs deliver major speedups.
Category: Example
What are risks of Mixed Precision Training?
Numerical issues like underflow/overflow, and subtle accuracy regressions if not validated.
Category: Risks
What competes with Mixed Precision Training?
Full Precision Training and Quantization Aware Training, each with different trade-offs.
Category: Alternatives
How to enable Mixed Precision Training in PyTorch?
Use autocast context and GradScaler, and validate results against FP32 baselines.
Category: HowTo