DataTerminology.comNEW
Also known as: Model Distillation, Teacher-Student Learning
A technique where a smaller student model learns to mimic a larger teacher model’s behavior using softened targets or representations.
DESCRIPTION
Knowledge distillation compresses models by training a smaller student to match a larger teacher’s outputs or intermediate representations. It uses soft targets with temperature scaling to convey dark knowledge. Distillation reduces latency and memory while preserving accuracy. Variants include offline, online, and self-distillation. It complements pruning and quantization for compound compression. Careful teacher selection, data coverage, and loss balancing are critical to success.
RELATED TERMS
Model Compression Student Model Teacher Model Temperature Scaling Soft Labels Fine-Tuning Calibration
COMPETING TERMS
RELATIONS
- Broader: Distillation
KNOWLEDGE AREAS
AI Data Science MLOps Mathematics
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Knowledge distillation
- arXiv - Original distillation paper by Hinton et al.
- PyTorch - Hands-on tutorial in PyTorch
- OpenVINO - OpenVINO guide to distillation
TOP QUESTIONS
Why use Knowledge Distillation for deployment?
It yields smaller, faster models with accuracy close to large teachers, making real-time and edge deployment feasible.
Category: Relevancy
What is temperature in Knowledge Distillation?
Temperature softens the teacher’s logits to reveal relative class probabilities, guiding the student more effectively than hard labels.
Category: Example
Can Knowledge Distillation be combined with quantization or pruning?
Yes. Distillation often restores accuracy lost by quantization or pruning, enabling higher compression ratios.
Category: Management
What are common mistakes in Knowledge Distillation?
Using mismatched architectures without tuning losses, neglecting data diversity, or setting an ineffective temperature can degrade results.
Category: Mistakes
What data do I need for Knowledge Distillation?
A dataset representative of deployment scenarios is essential; unlabeled data can be used since the teacher provides soft labels.
Category: Requirements
What are risks of Knowledge Distillation?
Teacher biases and errors propagate to the student; lack of transparency may complicate auditing.
Category: Risks
Is Knowledge Distillation useful for LLMs?
Yes. It transfers capabilities from large language models to smaller ones for memory- and latency-constrained environments.
Category: Opinion
How do I evaluate a distilled model?
Compare accuracy, latency, and memory against the teacher and a non-distilled student baseline on relevant benchmarks.
Category: Validation
What is the origin of Knowledge Distillation?
It was popularized by Hinton et al., highlighting soft targets to convey dark knowledge from large models.
Category: Origin