DataTerminology.comNEW
Also known as: Relative entropy
A measure of how one probability distribution diverges from another reference distribution.
DESCRIPTION
Kullback–Leibler Divergence quantifies the directed discrepancy between two probability distributions. It is non-symmetric and becomes zero when the distributions match almost everywhere. KL divergence underpins variational inference, information gain, and many ML losses. In practice it measures inefficiency of assuming Q when the true distribution is P. Its asymmetry motivates symmetric alternatives like Jensen–Shannon Divergence.
ALTERNATIVE DEFINITIONS
Wikipedia
In mathematical statistics, the Kullback–Leibler divergence (also called relative entropy) is a measure of how one probability distribution is different from a second, reference probability distribution.
Defines KL as relative entropy and highlights its directed nature.
TensorFlow
Computes Kullback-Leibler divergence loss between y_true and y_pred.
Frames KL as a practical loss function in deep learning APIs.
RELATED TERMS
Log Loss Entropy Mutual Information Total Variation Distance Variational Inference Maximum Likelihood Estimation F-divergence Information Gain
COMPETING TERMS
Jensen–Shannon Divergence Wasserstein Distance
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Kullback–Leibler divergence
- TensorFlow - API docs for KL divergence loss
- arXiv - VI tutorial using KL divergence
TOP QUESTIONS
Why is Kullback–Leibler Divergence asymmetric?
KL measures the expected log difference under one distribution versus another. Reversing roles changes the expectation, causing asymmetry.
Category: Theory
When should you use KL versus Jensen–Shannon Divergence?
Use KL for directed optimization like variational inference. Use Jensen–Shannon when a symmetric, bounded similarity is desired.
Category: Comparison
How does KL relate to maximum likelihood?
Maximizing likelihood is equivalent to minimizing KL divergence between empirical and model distributions.
Category: Theory
What is the risk of using KL in practice?
If distributions have disjoint support, KL can diverge to infinity, destabilizing optimization.
Category: Risk
How is KL used in variational autoencoders (VAEs)?
VAEs include a KL term to regularize the approximate posterior against a prior, promoting smooth latent spaces.
Category: Application
What’s the biggest misunderstanding about KL divergence?
Many treat it as a distance metric; it’s not symmetric and doesn’t satisfy the triangle inequality.
Category: Misunderstanding
Is KL always non-negative?
Yes, by Gibbs’ inequality, KL divergence is non-negative and zero only when distributions are equal almost everywhere.
Category: Theory
How do you compute KL for continuous distributions?
Use integrals of log-density ratios weighted by the true distribution; closed forms exist for some families (e.g., Gaussians).
Category: How-to
What is a practical alternative to KL for generative models?
Wasserstein distance can provide smoother gradients and better training stability in some settings.
Category: Alternatives