DataTerminology.comNEW
Also known as: 4-bit
Four-bit integer precision used to compress model parameters and speed inference with minimal accuracy loss when properly calibrated.
DESCRIPTION
INT4 quantization represents weights and sometimes activations using four-bit integers, shrinking memory and bandwidth needs. It can substantially increase throughput on hardware supporting low-precision arithmetic. Techniques like NF4 and per-channel scaling reduce accuracy loss. INT4 is popular for large language model serving and edge deployment where resources are constrained. Careful calibration and evaluation are essential to manage numerical errors.
ALTERNATIVE DEFINITIONS
Wikipedia
In computing, a nibble is a four-bit aggregation, or half an octet.
Defines the 4-bit unit underlying INT4 representation.
QLoRA
We introduce 4-bit NormalFloat (NF4), a data type that is information theoretically optimal for normally distributed weights.
Proposes NF4 datatype enabling effective 4-bit quantization for LLMs.
Hugging Face
4-bit quantization reduces memory usage and enables running larger models on limited hardware.
Practical guidance on benefits and trade-offs of 4-bit quantization.
RELATED TERMS
Quantization NF4 QLoRA Calibration Activation Quantization Weight Quantization Throughput Memory Footprint
COMPETING TERMS
KNOWLEDGE AREAS
AI Infrastructure MLOps Mathematics
TARGET AUDIENCE
AI Engineer MLOps Engineer Data Engineer Cloud Architect
EXTERNAL SOURCES
- Wikipedia - Nibble
- Hugging Face - Guide to 4-bit quantization
- arXiv - QLoRA introduces NF4
- NVIDIA - Low-precision inference support
TOP QUESTIONS
Why use INT4 for inference?
It cuts memory and bandwidth significantly, enabling faster and cheaper serving, especially for large models.
Category: Relevancy
How does INT4 affect accuracy?
With proper calibration and datatypes like NF4, accuracy loss can be modest, but task-specific evaluation is mandatory.
Category: Risks
What hardware supports INT4?
Some modern GPUs and specialized accelerators support 4-bit math or dequantization-efficient kernels.
Category: Requirements
How is INT4 applied to LLMs?
Weights are quantized to 4-bit with per-channel scales; activations may remain higher precision to preserve quality.
Category: Mechanism
How does INT4 compare to INT8?
INT4 offers higher compression and potential speedups but typically larger accuracy trade-offs than INT8.
Category: Comparison
What is NF4 in INT4 quantization?
A 4-bit floating-like datatype targeting normally distributed weights, improving quantization fidelity.
Category: Definition
What are common mistakes with INT4?
Skipping calibration, quantizing sensitive layers indiscriminately, and not testing on target distributions.
Category: Mistakes
Does INT4 help on CPU?
Potentially, if kernels and instruction sets support 4-bit operations or efficient dequantization.
Category: Opinion
Can I fine-tune quantized INT4 models?
Methods like QLoRA enable parameter-efficient finetuning using low-bit representations.
Category: Application