+ - 0:00:00
Notes for current slide
Notes for next slide

Boosting Decision Trees: Tuning

Dr. D’Agostino McGowan

1 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters

With bagging what could we tune?

2 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters

With bagging what could we tune?

  • The depth of the tree (tree_depth)
  • B, the number of bootstrapped training samples (the number of decision trees fit)
2 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters

With bagging what could we tune?

  • The depth of the tree (tree_depth)
  • B, the number of bootstrapped training samples (the number of decision trees fit)
  • It is more efficient to just pick something very large instead of tuning tree_depth
  • For B, you don't really risk overfitting if you pick something too big
2 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters

With bagging what could we tune?

  • The depth of the tree (tree_depth)
  • B, the number of bootstrapped training samples (the number of decision trees fit)
  • It is more efficient to just pick something very large instead of tuning tree_depth
  • For B, you don't really risk overfitting if you pick something too big

With random forest what could we tune?

2 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters

With bagging what could we tune?

  • The depth of the tree (tree_depth)
  • B, the number of bootstrapped training samples (the number of decision trees fit)
  • It is more efficient to just pick something very large instead of tuning tree_depth
  • For B, you don't really risk overfitting if you pick something too big

With random forest what could we tune?

  • The depth of the tree, B, and m the number of predictors to try
2 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters

With bagging what could we tune?

  • The depth of the tree (tree_depth)
  • B, the number of bootstrapped training samples (the number of decision trees fit)
  • It is more efficient to just pick something very large instead of tuning tree_depth
  • For B, you don't really risk overfitting if you pick something too big

With random forest what could we tune?

  • The depth of the tree, B, and m the number of predictors to try
  • The default is p, and this does pretty well
2 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • B the number of bootstraps
  • λ the shrinkage parameter
  • d the number of splits in each tree
3 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • Unlike bagging and random forest with boosting you can overfit if B is too large
4 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • Unlike bagging and random forest with boosting you can overfit if B is too large

    What do you think you can use to pick B?

4 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • Unlike bagging and random forest with boosting you can overfit if B is too large

    What do you think you can use to pick B?

  • Cross-validation, of course!
4 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • The shrinkage parameter λ controls the rate at which boosting learn
5 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • The shrinkage parameter λ controls the rate at which boosting learn
  • λ is a small, positive number, typically 0.01 or 0.001
5 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • The shrinkage parameter λ controls the rate at which boosting learn
  • λ is a small, positive number, typically 0.01 or 0.001
  • It depends on the problem, but typically a very small λ can require a very large B for good performance
5 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • The number of splits, d, in each tree controls the complexity of the boosted ensemble
6 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • The number of splits, d, in each tree controls the complexity of the boosted ensemble
  • Often d=1 is a good default
6 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • The number of splits, d, in each tree controls the complexity of the boosted ensemble
  • Often d=1 is a good default brace yourself for another tree pun!
6 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • The number of splits, d, in each tree controls the complexity of the boosted ensemble
  • Often d=1 is a good default brace yourself for another tree pun!
  • In this case we call the tree a stump meaning it just has a single split
6 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • The number of splits, d, in each tree controls the complexity of the boosted ensemble
  • Often d=1 is a good default brace yourself for another tree pun!
  • In this case we call the tree a stump meaning it just has a single split
  • This results in an additive model
6 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters for boosting

  • The number of splits, d, in each tree controls the complexity of the boosted ensemble
  • Often d=1 is a good default brace yourself for another tree pun!
  • In this case we call the tree a stump meaning it just has a single split
  • This results in an additive model
  • You can think of d as the interaction depth it controls the interaction order of the boosted model, since d splits can involve at most d variables
6 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani
7 / 7
Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Tuning parameters

With bagging what could we tune?

2 / 7
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, →, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow