DataTerminology.comNEW
A technique that prunes decision trees by minimizing a function of error and subtree complexity.
DESCRIPTION
Cost–complexity pruning removes branches from a decision tree to reduce overfitting by balancing training error with a penalty on tree size. The method evaluates subtrees using a parameter α (alpha) that trades off fit and complexity. Practitioners typically select α via cross-validation to maximize generalization. This approach is central to CART and widely available in ML libraries. It yields simpler, more interpretable models with better performance on unseen data.
ALTERNATIVE DEFINITIONS
scikit-learn
Minimal cost-complexity pruning is an algorithm for pruning decision trees to avoid overfitting. The complexity parameter alpha controls the trade-off between tree size and training error.
Focuses on alpha as a complexity parameter in practical implementation.
Wikipedia
Cost–complexity pruning finds a subtree that minimizes R_α(T) = R(T) + α|T|, where R(T) is error and |T| is the number of terminal nodes.
Explicitly presents the objective function used for pruning.
CMU
Cost-complexity pruning reduces overfitting by penalizing the size of the tree while balancing misclassification error.
Emphasizes overfitting control via a size penalty.
RELATED TERMS
Decision Tree Classification and Regression Trees Pruning Model Overfitting Regularization Cross-Validation Minimal Cost-Complexity Pruning Hyperparameter Tuning Alpha Bias-Variance Tradeoff
COMPETING TERMS
Reduced Error Pruning Pre-pruning
RELATIONS
- Broader: Pruning
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Pruning (decision trees)
- scikit-learn - Library docs for pruning parameter alpha
- CRAN rpart vignette - CART pruning in rpart
TOP QUESTIONS
Why use Cost-Complexity Pruning for decision trees?
It reduces overfitting, improving generalization by balancing error with model size. The result is a simpler, more interpretable tree.
Category: Relevancy
How is alpha chosen in Cost-Complexity Pruning?
Alpha is typically tuned via cross-validation to maximize validation performance and avoid overfitting.
Category: Requirements
What is the difference between Cost-Complexity Pruning and pre-pruning?
Pre-pruning constrains depth or leaf size during training; cost-complexity prunes after training using an explicit complexity penalty.
Category: Opinion
Can Cost-Complexity Pruning harm accuracy?
Excessive pruning can underfit. Monitor validation metrics and adjust alpha to find the best trade-off.
Category: Risk
Is Cost-Complexity Pruning available in scikit-learn?
Yes. DecisionTreeClassifier and DecisionTreeRegressor support pruning via the ccp_alpha parameter.
Category: Application
What is a common mistake when applying Cost-Complexity Pruning?
Tuning alpha on the training set alone. Use cross-validation to avoid optimistic bias.
Category: Mistakes
Does Cost-Complexity Pruning improve interpretability?
Often, yes. Smaller trees are easier to visualize and explain to stakeholders.
Category: Example
What metrics should guide Cost-Complexity Pruning?
Use validation loss or accuracy, and inspect learning curves to balance bias and variance.
Category: Management
How does Cost-Complexity Pruning compare to Reduced Error Pruning?
Reduced error removes nodes that do not reduce validation error; cost-complexity uses a formal penalty function for model size.
Category: Alternatives