DataTerminology.comNEW
Also known as: Relative entropy
A non-symmetric measure quantifying how one probability distribution differs from a reference distribution.
DESCRIPTION
KullbackāLeibler divergence measures the expected log difference between two distributions, central to information theory and Bayesian inference. It underpins variational inference via the ELBO objective and regularizes VAEs. KL is asymmetric and can be infinite, making alternatives like JensenāShannon and Wasserstein useful in some settings. It relates closely to cross-entropy and entropy.
ALTERNATIVE DEFINITIONS
Wikipedia
In probability theory and information theory, the KullbackāLeibler divergence (also called relative entropy) is a measure of how one probability distribution differs from a second, reference probability distribution.
Defines KL divergence with emphasis on information-theoretic context and asymmetry.
Distill
The KL divergence measures how one distribution is different from another.
Concise explanation in the context of variational autoencoders.
TensorFlow Probability
Computes the Kullback-Leibler divergence KL[pāq].
Developer-oriented API for KL divergence computation.
RELATED TERMS
Entropy Log Loss Mutual Information Variational Inference Probability Distribution Information Theory Variational Autoencoder Maximum Likelihood
COMPETING TERMS
JensenāShannon Divergence Wasserstein Distance
KNOWLEDGE AREAS
TARGET AUDIENCE
EXTERNAL SOURCES
- Wikipedia - KullbackāLeibler divergence
- TensorFlow Probability - API for KL divergence
- Distill - Intuition for KL in VAEs
- Kingma & Welling - VAE paper using KL regularization
TOP QUESTIONS
What is KL Divergence?
It measures how one probability distribution differs from a reference distribution, commonly used in information theory and Bayesian inference.
Category: Definition
Is KL Divergence symmetric?
No, KL is asymmetric (KL(pāq) ā KL(qāp)), so it is not a true distance metric.
Category: Misunderstanding
Where is KL Divergence used in machine learning?
It appears in variational inference (ELBO), VAEs, regularization terms, and model evaluation via distributional comparisons.
Category: Example
What are alternatives to KL Divergence?
JensenāShannon divergence offers symmetry; Wasserstein distance provides a different geometry for distributions.
Category: Alternatives
What risks arise when using KL Divergence?
KL can be infinite for non-overlapping support; careful modeling or alternative divergences may be needed.
Category: Risks
How does KL Divergence relate to cross-entropy?
Cross-entropy equals entropy plus KL; minimizing cross-entropy often implies minimizing KL to the true distribution.
Category: Relation
How is KL Divergence computed for continuous distributions?
It involves an expectation over the reference distribution of the log-density ratio, often approximated via sampling.
Category: HowTo
Who introduced KL Divergence?
Solomon Kullback and Richard Leibler introduced it in the 1950s within information theory.
Category: Origin
What is the biggest misunderstanding about KL Divergence?
Treating KL as a proper distance despite its asymmetry and failure to satisfy the triangle inequality.
Category: Misunderstanding
How do VAEs use KL Divergence?
VAEs include a KL term encouraging the approximate posterior to stay close to the prior, balancing reconstruction and regularization.
Category: Example