9.3: Conceptual Foundations

Gradient Boosting is grounded in the idea that even models with limited predictive capability, often referred to as weak learners, can be transformed into a powerful predictive system when combined strategically. Unlike methods that attempt to build a single perfect model in one step, Gradient Boosting takes an iterative approach. It starts with a simple baseline prediction, identifies where that prediction falls short, and then builds a new model specifically to correct those errors. This process is repeated many times, with each model improving upon the last by targeting the residual mistakes. The term gradient comes from the fact that the algorithm uses gradient descent, a mathematical optimization technique, to minimize the chosen loss function and guide the learning process. One of the most important distinctions between Gradient Boosting and other ensemble methods is its sequential nature (i.e., each step is informed by the results of the step before) making it more precise but also more sensitive to parameter tuning compared to parallel approaches like Random Forests.

The following core ideas form the foundation for understanding how Gradient Boosting works in practice:

• Weak Learners: Gradient Boosting typically uses shallow decision trees (stumps) as base learners. These models alone have limited predictive power, but when combined sequentially, they form a strong model.
• Sequential Learning: Each new model focuses on the residual errors of the previous one.
• Minimizing a Loss Function: Gradient Boosting uses gradient descent to minimize an objective function.
• Difference from Random Forest: Random Forest builds many trees in parallel and averages results, while Gradient Boosting builds trees sequentially, each improving on the last.