AI Foundations

Ensemble Methods

Ensemble methods combine the predictions of multiple models so the group is more accurate than any single member, mainly by reducing variance (bagging), reducing bias (boosting), or learning how to blend models (stacking).

What ensemble methods are

An ensemble method trains several predictors and combines their outputs into a single decision. The motivating observation is that a collection of models that each make different errors can correct one another: where one model is wrong, the others can outvote it. For classification the combination is usually a majority vote or an average of predicted class probabilities; for regression it is an average of the numerical predictions.

Three families dominate practice. Bagging trains many models in parallel on resampled copies of the data and averages them to cut variance. Boosting trains models in sequence, each one focused on the examples its predecessors got wrong, to cut bias. Stacking trains a separate meta-model that learns how best to weight the base models' predictions. These strategies are not mutually exclusive, and many production systems chain them together.

Ensembles trade computation and interpretability for accuracy. A single decision tree is easy to read; a forest of hundreds is not. The payoff is that for many real problems, especially structured tabular data, a well-tuned ensemble of trees remains the most accurate model available, and it is the default first choice in much applied machine learning and in competition leaderboards.

A base learner is the individual model; the ensemble is the combined predictor built from many base learners.
Combination rules differ by task: majority vote or probability averaging for classification, numeric averaging for regression.
Ensembles work best when base learners are accurate yet make uncorrelated errors, so diversity among members is as important as individual accuracy.
The cost is extra training and inference compute plus reduced interpretability compared with a single model.

The bias-variance reasoning

A model's expected error decomposes into bias (systematic error from a model too simple to capture the pattern), variance (sensitivity to the particular training sample), and irreducible noise. The two main ensemble families attack different terms of this decomposition, which is why they help for different reasons.

Averaging many predictors reduces variance. If the members were perfectly independent, averaging k of them would divide the variance by k. In practice the members are correlated because they share the same data and algorithm, so the reduction is smaller, but it is still substantial. This is exactly why bagging pairs well with high-variance, low-bias base learners such as deep, unpruned decision trees: the averaging cancels their instability while preserving their flexibility.

Boosting instead reduces bias. It builds an additive model in stages, adding a new weak learner at each step that corrects the residual errors of the current ensemble. A weak learner is one only slightly better than chance, such as a shallow tree, which on its own has high bias. Summing many of them produces a low-bias model. Because boosting keeps fitting the training data more closely, it can overfit if run too long, so the number of stages and a learning rate must be tuned.

Var( (1/k) Σ_{i=1}^{k} h_i(x) ) = ρ·σ² + ((1 - ρ)/k)·σ²

Variance of an average of k base learners, each with variance σ² and pairwise correlation ρ. Lowering ρ (decorrelating the models) and raising k both shrink the second term, which is the source of bagging's gains.

Bagging targets variance by averaging independent-ish predictors; gains grow as member errors become less correlated.
Boosting targets bias by sequentially adding learners that fit the previous ensemble's residuals.
Bagging suits high-variance base learners (deep trees); boosting suits high-bias base learners (shallow stumps).
Boosting can overfit with too many rounds; bagging is comparatively resistant to overfitting from added members.

Bagging and Random Forests

Bagging, short for bootstrap aggregating, was introduced by Leo Breiman in 1996. A bootstrap sample is drawn by sampling the training set with replacement to the same size, so roughly 63 percent of the original points appear in each sample and the rest are duplicates or absent. A separate model is trained on each bootstrap sample, and their predictions are averaged or voted. The points left out of a given sample form an out-of-bag set that gives a free validation estimate without a held-out split.

Random Forests, introduced by Breiman in 2001, extend bagging to decision trees with one extra source of randomness: at each split the tree may only consider a random subset of the features. This decorrelates the trees, since they can no longer all latch onto the same dominant feature, and lower correlation directly improves the averaging as the variance formula shows. Random Forests need little tuning, give feature-importance estimates, and are a strong baseline on many problems.

Bagging is embarrassingly parallel: every base model trains independently, so the work spreads across cores or machines with no coordination. The main limitation is that it reduces variance but does little for bias, so if the base learner systematically underfits, a bagged ensemble of it will underfit too.

python

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=20, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)

clf = RandomForestClassifier(n_estimators=300, max_features="sqrt", random_state=0)
clf.fit(X_tr, y_tr)
print(clf.score(X_te, y_te))

A Random Forest in scikit-learn. n_estimators sets the number of bagged trees; max_features='sqrt' limits each split to a random feature subset, which is the decorrelation step that distinguishes a forest from plain bagged trees.

Bootstrap aggregating (Breiman, 1996) trains each member on a with-replacement resample of the data.
Out-of-bag samples, the points excluded from each bootstrap, provide a built-in validation estimate.
Random Forests (Breiman, 2001) add per-split random feature subsets to decorrelate the trees and sharpen the variance reduction.
Bagging parallelizes trivially but cannot fix a base learner's bias.

Boosting: AdaBoost, gradient boosting, and modern libraries

Boosting builds the ensemble one learner at a time, each correcting the mistakes of the running total. AdaBoost, introduced by Yoav Freund and Robert Schapire in 1997, does this by reweighting the training examples: after each weak learner, misclassified points get larger weights so the next learner concentrates on them, and the final prediction is a weighted vote in which more accurate learners count for more.

Gradient boosting, formalized by Jerome Friedman in 2001, generalizes the idea to any differentiable loss. It treats boosting as gradient descent in function space: at each stage it fits a new weak learner to the negative gradient (the pseudo-residuals) of the loss with respect to the current prediction, then adds it to the model scaled by a step size. With squared-error loss the pseudo-residuals are just the ordinary residuals, which recovers the intuition of repeatedly fitting what is left over.

Modern gradient-boosting libraries make this fast and accurate at scale. XGBoost, from Tianqi Chen and Carlos Guestrin in 2016, added a regularized objective, a sparsity-aware split finder, and cache-aware engineering. LightGBM uses histogram-based splits and leaf-wise growth for speed on large data, and CatBoost adds native handling of categorical features with ordered boosting to reduce target leakage. These three are the workhorses of tabular machine learning today.

F_m(x) = F_{m-1}(x) + γ_m · h_m(x)

The additive form of boosting. The ensemble at stage m is the previous ensemble plus a new weak learner h_m fitted to the current residuals or pseudo-gradients, scaled by a step size γ_m (the learning rate).

python

import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=20, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)

model = xgb.XGBClassifier(n_estimators=400, learning_rate=0.05, max_depth=4)
model.fit(X_tr, y_tr)
preds = model.predict(X_te)

Gradient-boosted trees with XGBoost. learning_rate is the shrinkage step γ; pairing a small rate with more n_estimators is a common way to trade training time for generalization.

AdaBoost (Freund and Schapire, 1997) reweights misclassified examples so each learner focuses on prior errors.
Gradient boosting (Friedman, 2001) fits each new learner to the negative gradient of any differentiable loss.
A learning rate (shrinkage) scales each added learner; smaller rates with more rounds usually generalize better.
XGBoost (2016), LightGBM, and CatBoost are the dominant production implementations of gradient-boosted trees.

Stacking and why tree ensembles win on tabular data

Stacking, or stacked generalization, combines different model types with a learned blender. Several diverse base models (say a forest, a boosted ensemble, and a linear model) each produce predictions, and a meta-learner is trained to map those predictions to the final output. To avoid the meta-learner simply memorizing base models that have already seen the data, the base predictions are generated out-of-fold via cross-validation. Stacking can squeeze out extra accuracy when the base models have genuinely different strengths, at the cost of a more complex pipeline.

On tabular data, ensembles of trees frequently beat neural networks. The 2022 study by Grinsztajn and colleagues benchmarked tree-based models and deep networks across many datasets and found tree ensembles remained state of the art on medium-sized tabular problems even after tuning. Their analysis attributes this to inductive biases: trees handle heterogeneous, unscaled, mixed-type features and uninformative columns gracefully, whereas neural networks were designed around the smoothness and spatial structure of images, audio, and text.

The practical reading is that the right ensemble depends on the data. For images, audio, and language, deep networks dominate. For structured rows and columns, gradient-boosted trees and Random Forests are usually the first thing to try, and stacking a few strong models on top is a reliable way to gain the last few points of accuracy.

Stacking trains a meta-learner on the out-of-fold predictions of diverse base models to learn the best blend.
Out-of-fold prediction is essential in stacking to prevent the meta-learner from trusting overfit base models.
Tree ensembles remain state of the art on much medium-sized tabular data, even versus tuned deep networks (Grinsztajn et al., 2022).
Deep learning still wins on perceptual data (images, audio, text); tree ensembles win on heterogeneous tabular features.

Key takeaways

Ensemble methods combine multiple models so the group beats any single member by reducing variance, bias, or both.
Bagging (Breiman, 1996) trains members in parallel on bootstrap samples and averages them to cut variance; Random Forests add random feature subsets to decorrelate the trees.
Boosting builds members sequentially, each correcting prior errors: AdaBoost (1997) reweights examples, gradient boosting (Friedman, 2001) fits the loss gradient.
XGBoost, LightGBM, and CatBoost are the dominant modern gradient-boosting libraries for tabular machine learning.
Stacking trains a meta-learner on out-of-fold base-model predictions to learn how best to blend diverse models.
On medium-sized tabular data, tree ensembles still outperform tuned deep networks because of their inductive biases (Grinsztajn et al., 2022).

Frequently asked questions

Bagging trains models independently in parallel on bootstrap resamples and averages them, which mainly reduces variance. Boosting trains models sequentially, each focused on the errors of the previous ones, which mainly reduces bias. Bagging suits high-variance learners like deep trees, while boosting suits high-bias learners like shallow stumps.

If individual models make different, uncorrelated errors, averaging or voting lets correct members outweigh the mistaken ones. Averaging k roughly independent predictors divides their variance by about k, and boosting drives down bias by repeatedly fitting what the current ensemble still gets wrong. The benefit depends on the members being both reasonably accurate and diverse.

A Random Forest is bagging applied to decision trees plus one extra trick: at each split it considers only a random subset of features. That feature randomness decorrelates the trees, which improves the variance reduction beyond what plain bagged trees achieve. So every Random Forest uses bagging, but not all bagging is a Random Forest.

A 2022 benchmark by Grinsztajn and colleagues found tree-based models remained state of the art on medium-sized tabular datasets even after tuning. The reason is inductive bias: trees handle heterogeneous, unscaled, mixed-type features and ignore uninformative columns easily, while neural networks were built around the smoothness and spatial structure of images, audio, and text.

The learning rate, also called shrinkage, scales the contribution of each newly added weak learner in the additive model F_m(x) = F_{m-1}(x) + γ_m h_m(x). Smaller learning rates require more boosting rounds but usually generalize better, so it is tuned together with the number of estimators. It is one of the most important hyperparameters in XGBoost, LightGBM, and CatBoost.

Stacking helps when several genuinely different model types each capture different aspects of the data, since a meta-learner can then blend their strengths. It is most useful for squeezing out the last few points of accuracy, for example in competitions, after a strong gradient-boosted model is already in place. The cost is a more complex pipeline and the need for careful out-of-fold prediction to avoid leakage.

Put the idea into practice

MemX is an AI memory agent built on these ideas: store anything, skip the folders, and find it again by asking in plain English.

Try MemX Free