DataTerminology.comNEW
A technique that transfers knowledge from a larger teacher model to a smaller student for efficiency.
DESCRIPTION
Knowledge distillation trains a compact student model to mimic a powerful teacher’s outputs or intermediate representations. Soft targets and temperature scaling guide learning beyond hard labels. Distillation reduces latency, memory, and cost while retaining accuracy for deployment. Variants include offline, online, and self-distillation. It complements compression methods like pruning and quantization, and is widely used in production-serving environments.
ALTERNATIVE DEFINITIONS
Hinton, Vinyals, Dean
We show that the knowledge in a large model can be transferred to a small model by training the small model to mimic the outputs of the large model.
Introduces core teacher-student paradigm and soft targets for transfer.
Wikipedia
Knowledge distillation is the process of transferring knowledge from a large model to a smaller one, typically by training the small model to mimic the large model.
Concise definition emphasizing mimicry-driven training.
RELATED TERMS
Model Compression Teacher-Student Fine-Tuning Soft Labels Temperature Scaling Hint Training Ensembling Superclass
COMPETING TERMS
RELATIONS
- Narrower: Defensive Distillation Knowledge Distillation
KNOWLEDGE AREAS
TARGET AUDIENCE
AI Engineer Data Scientist MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Knowledge distillation
- Hinton et al. - Foundational paper introducing distillation
- Hugging Face - Practical guide to distillation
TOP QUESTIONS
What is Knowledge Distillation?
Distillation transfers a teacher model’s behavior to a smaller student, retaining performance while reducing compute and memory.
Category: Definition
Why use Distillation?
It enables efficient deployment, faster inference, and lower costs while preserving accuracy on key tasks.
Category: Relevancy
How does Distillation work?
Students train on soft targets produced by teachers, often using temperature scaling and sometimes intermediate feature matching.
Category: HowTo
What are common pitfalls in Distillation?
Poor teacher quality, mismatched temperature, and insufficient student capacity can degrade results.
Category: Mistakes
How does Distillation compare to Pruning and Quantization?
Distillation transfers knowledge; pruning removes weights; quantization reduces precision. Combined, they maximize efficiency.
Category: Comparison
Can Distillation improve generalization?
Yes, soft targets can regularize learning and capture teacher-induced class relationships that improve student generalization.
Category: Opinion
Is Distillation applicable to LLMs?
Yes, it is used to create smaller, faster LLMs, often combined with LoRA and dataset curation.
Category: UseCase
How to evaluate a distilled model?
Use task benchmarks, latency and memory profiling, and robustness tests against domain shifts.
Category: Measurement
Where to start with Distillation?
Begin with a strong teacher, tune temperature, and validate student performance on representative datasets.
Category: GettingStarted