DataTerminology.comNEW
Also known as: Entropy Reduction
A splitting criterion measuring the reduction in entropy when a dataset is partitioned on an attribute, commonly used in decision tree learning.
DESCRIPTION
Information gain quantifies how much uncertainty decreases after splitting data by a feature. In decision trees, it helps select attributes that best separate classes. It is computed as parent entropy minus the weighted entropy of child nodes. While effective, it favors attributes with many distinct values, motivating normalized variants like gain ratio. Information gain also relates to mutual information in information theory. It is widely used in ID3 and C4.5 algorithms.
ALTERNATIVE DEFINITIONS
Wikipedia
Information gain is the reduction in entropy achieved because of partitioning the examples in a training dataset according to an attribute.
Directly ties information gain to entropy reduction during attribute-based splits in decision trees.
Princeton University
Information gain measures the expected reduction in entropy from partitioning the data according to a given attribute.
Academic lecture notes emphasizing expected entropy reduction for attribute selection.
RELATED TERMS
Entropy Decision Tree ID3 C4.5 Algorithm Mutual Information Feature Reduction Log Loss Random Forest
COMPETING TERMS
Gini Impurity Gain Ratio
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer Data Analyst
EXTERNAL SOURCES
- Wikipedia - Information gain in decision trees
- scikit-learn - Decision tree criteria overview
- MIT Press - Mitchell’s book covering ID3
TOP QUESTIONS
Why is Information Gain used in decision trees?
It selects features that best reduce class uncertainty, leading to purer splits and more accurate trees.
Category: Relevancy
What is a common pitfall of Information Gain?
It biases toward attributes with many unique values. Gain ratio or pruning mitigates this bias.
Category: Mistakes
How is Information Gain computed?
Subtract the weighted sum of child entropies from the parent entropy for a given split.
Category: Requirements
When should I prefer Gini Impurity over Information Gain?
Gini is computationally simpler and works well in CART; performance differences are often small in practice.
Category: Opinion
How does Information Gain relate to mutual information?
They’re closely related; information gain in trees is a form of mutual information between feature and target.
Category: Misunderstanding
Does Information Gain work for regression trees?
No. Regression trees use variance reduction or similar continuous criteria rather than entropy-based metrics.
Category: Misunderstanding
How does Information Gain affect overfitting?
Greedy maximization can overfit; use pruning, min-samples, or depth limits to control complexity.
Category: Risks
Where else is Information Gain used?
Feature selection for text classification and filtering is common, ranking terms by information gain.
Category: Example
Is Information Gain robust to class imbalance?
It can be skewed by imbalance. Consider class weighting or alternative metrics in imbalanced settings.
Category: Risks