🔢BF16

NEW

Also known as: BFloat16

A 16-bit floating-point format with eight exponent bits and seven significand bits that maintains dynamic range suited for deep learning.

DESCRIPTION

bfloat16 (BF16) trades mantissa precision for a wider exponent range, matching FP32’s range to reduce underflow/overflow in deep learning. It accelerates training and inference on TPUs and modern GPUs with minimal accuracy loss compared to FP32. Mixed precision often pairs BF16 with selective FP32 accumulation for stability. Compared to FP16, BF16 is more numerically robust for gradients. Hardware support and libraries have standardized BF16 workflows. It is widely used in large-scale transformer training. BF16 reduces memory bandwidth and footprint, enabling larger batch sizes.

ALTERNATIVE DEFINITIONS

Wikipedia

bfloat16 (brain floating point) is a 16-bit floating-point representation consisting of 1 sign bit, 8 exponent bits, and 7 bits of significand.

States exact bit layout and origin of the name.

Source

Google Cloud

bfloat16 is a 16-bit floating point representation that preserves the dynamic range of 32-bit floating point numbers.

Emphasizes matching FP32 dynamic range for stability.

Source

RELATED TERMS

Mixed Precision Training Tensor Cores Quantization IEEE 754 Automatic Mixed Precision Loss Scaling Graphics Processing Unit TPU

COMPETING TERMS

FP16 FP32

RELATIONS

• Broader: IEEE 754

KNOWLEDGE AREAS

AI Mathematics Infrastructure

TARGET AUDIENCE

AI Engineer Data Scientist MLOps Engineer Cloud Architect

EXTERNAL SOURCES

• Wikipedia - Bfloat16 floating-point format
• Google Cloud - TPU docs explaining BF16
• NVIDIA - Mixed precision guidance

TOP QUESTIONS

Why use BF16 for deep learning?

It preserves FP32’s dynamic range, reducing numerical issues while cutting memory and boosting throughput on supported hardware.

Category: Relevancy

How does BF16 compare to FP16?

BF16 has a wider exponent and smaller mantissa than FP16, making it more robust to underflow/overflow in training.

Category: Comparison

When should I prefer FP32 over BF16?

Use FP32 for highly sensitive computations or where hardware lacks optimized BF16 support. Accumulations may still use FP32.

Category: Guidance

Does BF16 hurt model accuracy?

Typically minimal impact, especially with mixed precision and proper loss scaling. Validate on your specific task.

Category: Performance

What hardware supports BF16?

Google TPUs and recent NVIDIA/AMD GPUs support BF16 natively. Check device compute capability.

Category: Infrastructure

How does BF16 affect training speed?

It often increases throughput and allows larger batch sizes, improving utilization and training times.

Category: Scalability

Is BF16 a standard like IEEE FP16?

BF16 is widely adopted but not an IEEE-754 standard type; it is vendor-supported.

Category: Standard

Can I mix BF16 with FP32?

Yes, mixed precision uses BF16 for compute-heavy ops and FP32 for accumulations to preserve stability.

Category: Practice

Does BF16 help inference?

Yes, it reduces memory bandwidth and increases throughput, though accuracy must be verified for each model.

Category: Inference