DataTerminology.comNEW
Also known as: Gradient Tree Boosting, GBM
Ensemble method that sequentially adds weak learners to minimize loss via gradient-based optimization.
DESCRIPTION
Gradient boosting builds an additive model by fitting new learners to the residuals of prior learners. Typically, decision trees are used as weak learners. A learning rate and tree depth control regularization to prevent overfitting. It supports arbitrary differentiable loss functions for regression, classification, and ranking. Modern implementations like XGBoost, LightGBM, and CatBoost add efficiency and advanced regularization. It remains competitive for tabular data, often outperforming deep learning.
ALTERNATIVE DEFINITIONS
Wikipedia
Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees.
Highlights ensemble of weak learners, often decision trees.
scikit-learn
Gradient Tree Boosting builds an additive model in a forward stage-wise fashion; it allows for the optimization of arbitrary differentiable loss functions.
Emphasizes forward stage-wise optimization of differentiable losses.
XGBoost
Gradient boosting constructs new models to predict the residuals or errors of prior models and then adds them together to make the final prediction.
Focuses on residual fitting and additive model construction.
RELATED TERMS
XGBoost LightGBM CatBoost Boosting Decision Tree Ensemble Learning Learning Rate Regularization Stacked Generalization
COMPETING TERMS
RELATIONS
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Gradient boosting
- scikit-learn - Official gradient boosting docs
- XGBoost - Library documentation and guides
- LightGBM - Efficient gradient boosting docs
- CatBoost - Gradient boosting with categorical features
TOP QUESTIONS
Why is Gradient Boosting effective on tabular data?
It captures complex interactions and nonlinearities with decision trees, while regularization controls overfitting. Advanced implementations optimize speed and generalization.
Category: Relevancy
How does Gradient Boosting minimize loss?
It fits each new weak learner to the gradient or residuals of the current model, reducing the objective function iteratively.
Category: Definition
What is the role of learning rate in Gradient Boosting?
The learning rate scales each tree’s contribution, trading convergence speed for generalization. Lower rates often require more trees but improve robustness.
Category: Concept
How does Gradient Boosting compare to Random Forests?
Gradient Boosting builds trees sequentially to correct errors; Random Forests build trees independently using bagging. Boosting often achieves higher accuracy but risks overfitting.
Category: Distinction
What are common mistakes with Gradient Boosting?
Overly deep trees, high learning rates, and insufficient regularization lead to overfitting. Ignoring early stopping and validation degrades performance.
Category: Mistakes
Which libraries implement Gradient Boosting?
Popular libraries include XGBoost, LightGBM, CatBoost, and scikit-learn’s GradientBoosting.
Category: Overview
What losses are used in Gradient Boosting?
Common losses include least squares for regression, logistic loss for classification, and specialized ranking losses.
Category: Example
How is Gradient Boosting tuned?
Key hyperparameters include learning rate, number of estimators, tree depth, subsampling, and regularization terms like L1/L2.
Category: Best practices
Is Gradient Boosting still relevant with deep learning?
Yes. On structured, tabular data, Gradient Boosting often outperforms deep models and trains faster.
Category: Opinion