Boosting

• Like bagging, boosting is an approach that can be applied to many statistical learning methods
• We will discuss how to use boosting for decision trees

Boosting

• Bagging involves:
  • resampling from the original training data to make many bootstrapped training data sets
  • fitting a separate decision tree to each bootstrapped training data set
  • combining all trees to make one predictive model
• ☝️ Note, each tree is built on a bootstrap dataset, independent of the other trees
• Boosting is similar, except the trees are grown sequentially, using information from the previously grown trees

Boosting algorithm for regression trees

Step 1

• Set $\hat{f}\left(x\right)=0$ and $r_i=y_i$ for all $i$ in the training set

Boosting algorithm for regression trees

Step 2 For $b=1,2,\dots ,B$ repeat:

• Fit a tree ${\hat{f}}^b$ with $d$ splits ( $d$ + 1 terminal nodes) to the training data ( $X,r$ )
• Update $\hat{f}$ by adding in a shrunken version of the new tree:

$$\hat{f}\left(x\right)\leftarrow \hat{f}\left(x\right)+\lambda {\hat{f}}^b\left(x\right)$$

• Update the residuals:

$$r_i\leftarrow r_i-\lambda {\hat{f}}^b\left(x_i\right)$$

Boosting algorithm for regression trees

Step 3

• Output the boosted model

$$\hat{f}\left(x\right)=\sum _{b=1}^B\lambda {\hat{f}}^b\left(x\right)$$

Big picture

• Given the current model, we are fitting a decision tree to the residuals
• We then add this new decision tree into the fitted function to update the residuals
• Each of these trees can be small (just a few terminal nodes), determined by $d$
• Instead of fitting a single large decision tree, which could result in overfitting, boosting learns slowly

Big Picture

• By fitting small trees to the residuals we slowly improve $\hat{f}$ in areas where it does not perform well
• The shrinkage parameter $\lambda$ slows the process down even more allowing more and different shaped trees to try to minimize those residuals

Boosting for classification

• Boosting for classification is similar, but a bit more complex
• tidymodels will handle this for us, but if you are interested in learning more, you can check out Chapter 10 of Elements of Statistical Learning