7.3: Regression Trees

Introduction to Regression Trees

Regression trees are a type of decision tree algorithm used when the target (dependent) variable is continuous rather than categorical. Instead of classifying observations into discrete categories, regression trees predict a numerical outcome by learning decision rules from input features. At each internal node, the dataset is split based on a condition that best minimizes error—typically using criteria like minimizing the sum of squared deviations. Each terminal (leaf) node represents a predicted numeric value that corresponds to the average of the target variable for observations reaching that node. Regression trees are intuitive, visual, and capable of capturing non-linear relationships in the data.

Differences Between Regression and Classification Trees

Differences Between Regression and Classification Trees

Feature	Classification Trees	Regression Trees
Target Variable	Categorical	Continuous
Prediction Output	Class label (e.g., Yes/No)	Numeric value
Splitting Criteria	Gini Index, Entropy	Mean Squared Error, MAE
Evaluation Metrics	Accuracy, Sensitivity, Specificity	RMSE, MAE, R-squared
Example Use Case	Churn prediction	Revenue prediction

Applications of Regression Trees

Regression trees are useful in a variety of business and analytical scenarios where the outcome variable is numeric. For instance, marketers might predict customer lifetime value, sales teams might estimate future sales based on historical data, real estate agents can assess property prices based on location and features, and logistics managers may forecast delivery times depending on route and traffic patterns. These models help organizations make informed, data-driven decisions with a visual and interpretable output.

Image of a Regression Tree and Interpretation

In a regression tree, each node represents a split that attempts to reduce variance in the dependent variable. The branches reflect the conditions for the splits, and the leaf nodes show the predicted numeric values.

Consider the following hypothetical business scenario:

A subscription-based telecommunications company wants to better understand the factors influencing customer revenue so it can more accurately forecast income and tailor pricing strategies. Using historical customer data, the company built a regression tree to predict monthly revenue based on two key variables: customer tenure and monthly charges. The model first splits customers by tenure, recognizing that newer customers often have different spending patterns compared to long-term subscribers. Among those with less than 12 months of tenure, monthly charges are the next most important factor, with lower-charged customers predicting an average revenue of $55 and higher-charged customers predicting $72. Longer-tenure customers, regardless of monthly charges, show higher predicted revenue at $95. This analysis helps the company identify high-value segments, adjust retention offers for newer customers, and fine-tune pricing models to maximize revenue potential.

Following is an illustration of the regression tree based from the example above.

Metrics to Evaluate Regression Trees

To evaluate how well a regression tree performs, analysts commonly use the following metrics:

• RMSE (Root Mean Squared Error): Measures the square root of the average squared differences between predicted and actual values.
• MAE (Mean Absolute Error): Takes the average of the absolute errors without squaring them.
• R-squared: Indicates the proportion of variance in the dependent variable explained by the model.

These metrics help determine how closely the model’s predictions match the actual values.

Splitting Criteria for Regression Trees

Regression trees use variance reduction methods to determine the best split. At each node, the algorithm evaluates which split yields the greatest decrease in variability of the target variable. This is typically assessed using Mean Squared Error (MSE), which ensures that the selected split results in subsets of data with minimal variation around the mean.

Advantages and Disadvantages

Advantages:

• Easy to interpret and visualize
• Can model non-linear relationships
• No need for data normalization or scaling
• Suitable for both small and large datasets

Disadvantages:

• Prone to overfitting unless pruned or regularized
• Sensitive to minor data changes
• May underperform compared to ensemble methods like Random Forests

Summary

Regression trees are powerful yet intuitive tools for predicting numeric outcomes using decision rule logic. They are particularly valuable when transparency and interpretability are required. However, they may suffer from instability and limited accuracy when used alone.

To overcome these issues, ensemble methods like Random Forests build multiple trees and aggregate their predictions for more robust results. In Chapter 8, we will explore Random Forests in detail and see how they can improve performance across both regression and classification tasks.