DataTerminology.comNEW
Also known as: Bootstrap Aggregating
Ensemble learning technique that trains models on bootstrap samples and aggregates predictions to reduce variance and improve accuracy.
DESCRIPTION
Bagging (bootstrap aggregating) builds multiple models on different bootstrap-resampled subsets of the training data and combines their predictions by averaging or voting. It reduces variance, mitigates overfitting, and improves stability, especially for high-variance learners like decision trees. Out-of-bag estimates provide an internal validation of performance without a separate holdout set. Random Forest generalizes bagging by also randomizing feature selection. Bagging is simple, parallelizable, and effective when model variance dominates bias.
ALTERNATIVE DEFINITIONS
scikit-learn
A Bagging classifier is an ensemble meta-estimator that fits base classifiers each on random subsets of the original dataset and then aggregates their individual predictions (either by voting or by averaging) to form a final prediction.
Emphasizes practical implementation details and aggregation by voting/averaging.
Wikipedia
Bootstrap aggregating, also called bagging, is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regression.
Highlights stability and accuracy improvements across classification and regression.
Leo Breiman (Machine Learning, 1996)
Bagging predictors is a method for generating multiple versions of a predictor and using these to get an aggregated predictor.
Original formulation by the inventor; focuses on aggregation of multiple predictor versions.
RELATED TERMS
Ensemble Learning Random Forest Subsampling Variance Reduction Out-of-Bag Error Decision Tree Cross-Validation
COMPETING TERMS
RELATIONS
- Broader: Ensemble Learning
KNOWLEDGE AREAS
TARGET AUDIENCE
Data Scientist AI Engineer MLOps Engineer
EXTERNAL SOURCES
- Wikipedia - Bootstrap aggregating
- scikit-learn - User guide for bagging in scikit-learn
- Machine Learning (Springer) - Breiman’s original bagging paper
- Carnegie Mellon University - Academic slides on bagging
TOP QUESTIONS
How does Bagging reduce variance?
By training models on different bootstrap samples and averaging their outputs, Bagging cancels uncorrelated errors, reducing variance without significantly increasing bias.
Category: Concept
When should I use Bagging instead of Boosting?
Use Bagging when your base learners have high variance and low bias, like decision trees; Boosting is often better when reducing bias is the main concern.
Category: Decision
What is out-of-bag error in Bagging?
Out-of-bag error estimates performance using samples not included in each bootstrap, providing a built-in validation without a separate holdout set.
Category: Evaluation
Is Random Forest just Bagging?
Random Forest extends Bagging by also randomizing feature selection at each split, further decorrelating trees and improving ensemble performance.
Category: Comparison
What base learners work well with Bagging?
High-variance models like unpruned decision trees benefit most, but Bagging can be applied to many learners including regressors and classifiers.
Category: Practice
Does Bagging help with class imbalance?
Bagging alone is not designed for imbalance; combining it with stratified sampling or class weighting can improve results on imbalanced data.
Category: Risk
How many estimators should Bagging use?
Performance typically stabilizes as estimators increase; start with 50–200 and tune based on validation error and compute budget.
Category: Requirements
What’s the computational cost of Bagging?
It increases training cost linearly with the number of estimators but is highly parallelizable across CPUs or nodes.
Category: Tradeoff
Why is Bagging still relevant in the era of deep learning?
It remains effective for tabular data and tree-based methods, where ensembles often outperform deep models with less tuning and compute.
Category: Relevancy