DataTerminology.comNEW
Also known as: MI
A measure of shared information between two variables.
DESCRIPTION
Mutual Information quantifies the dependency between random variables by comparing joint and marginal entropies. It detects arbitrary nonlinear relationships, unlike Pearson correlation which captures only linear dependence. MI is widely used in feature selection, image registration, clustering, and information-theoretic analyses. It is non-negative and equals zero if and only if variables are independent. Estimation can be challenging and often relies on binning, k-nearest neighbors, or kernel methods.
ALTERNATIVE DEFINITIONS
Wikipedia
In probability theory and information theory, the mutual information (MI) of two random variables is a measure of the mutual dependence between the two variables.
Explicitly ties MI to mutual dependence in information theory.
scikit-learn
Mutual information between two random variables is a non-negative value, which measures the dependency between the variables.
Highlights non-negativity and use in feature selection.
RELATED TERMS
Entropy Conditional Entropy Kullback–Leibler Divergence Information Gain Information Theory Feature Reduction Log Loss Total Correlation
COMPETING TERMS
Pearson Correlation Coefficient Distance Correlation
RELATIONS
- Broader: Information
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Mutual information
- scikit-learn - Guidance on MI for feature selection
- NeurIPS - Foundational paper on MI estimation
TOP QUESTIONS
Why use Mutual Information instead of correlation?
Mutual Information detects arbitrary nonlinear dependencies, while Pearson correlation captures only linear relationships. MI is zero only under independence, making it more general.
Category: Comparison
How is Mutual Information estimated in practice?
Common approaches include discretization, k-nearest neighbor estimators, kernel density estimation, and entropy plug-in methods. Choice affects bias-variance trade-offs.
Category: How-to
What does MI = 0 imply?
It implies statistical independence between the variables under the assumed distributions and estimation method.
Category: Interpretation
What are common pitfalls when using MI?
Discretization artifacts, undersampling, and biased estimators can mislead. Always validate with sensitivity analysis and robust estimation.
Category: Mistakes
How is Mutual Information used in feature selection?
It ranks features by dependency with the target. Extensions like mRMR balance relevance and redundancy among selected features.
Category: Application
Is Mutual Information symmetric?
Yes, MI(X;Y) equals MI(Y;X).
Category: Definition
What units does Mutual Information use?
Units are bits when logarithms are base 2, or nats when base e.
Category: Basics
How does MI relate to entropy?
MI equals the sum of entropies minus the joint entropy: I(X;Y) = H(X) + H(Y) − H(X,Y).
Category: Theory
When is MI preferred for clustering?
Normalized Mutual Information is widely used to compare clustering results against ground truth due to scale invariance.
Category: Application