Non-linearity

Non-linearityDr. D’Agostino McGowan1 / 22

Non-linear relationships

What have we used so far to deal with non-linear relationships?

2 / 22

Non-linear relationships

What have we used so far to deal with non-linear relationships?

Hint: What did you use in Lab 03?

2 / 22

Non-linear relationships

What have we used so far to deal with non-linear relationships?

Hint: What did you use in Lab 03?
Polynomials!

2 / 22

Polynomials

$y_{i} = β_{0} + β_{1} x_{i} + β_{2} x_{i}^{2} + β_{3} x_{i}^{3} \dots + β_{d} x_{i}^{d} + ϵ_{i}$

3 / 22

Polynomials

$y_{i} = β_{0} + β_{1} x_{i} + β_{2} x_{i}^{2} + β_{3} x_{i}^{3} \dots + β_{d} x_{i}^{d} + ϵ_{i}$

This is data from the Columbia World Fertility Survey (1975-76) to examine household compositions

3 / 22

Polynomials

$y_{i} = β_{0} + β_{1} x_{i} + β_{2} x_{i}^{2} + β_{3} x_{i}^{3} \dots + β_{d} x_{i}^{d} + ϵ_{i}$

This is data from the Columbia World Fertility Survey (1975-76) to examine household compositions
Fit here with a 4th degree polynomial

3 / 22

Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

How is it done?New variables are created ( X1=XX1=X, X2=X2X2=X2, X3=X3X3=X3, etc) and treated as multiple linear regression
4 / 22

How is it done?

New variables are created ( $X_{1} = X$ , $X_{2} = X^{2}$ , $X_{3} = X^{3}$ , etc) and treated as multiple linear regression
We are not interested in the individual coefficients, we are interested in how a specific $x$ value behaves

$\hat{f} (x_{0}) = {\hat{β}}_{0} + {\hat{β}}_{1} x_{0} + {\hat{β}}_{2} x_{0}^{2} + {\hat{β}}_{3} x_{0}^{3} + {\hat{β}}_{4} x_{0}^{4}$

4 / 22

How is it done?

New variables are created ( $X_{1} = X$ , $X_{2} = X^{2}$ , $X_{3} = X^{3}$ , etc) and treated as multiple linear regression
We are not interested in the individual coefficients, we are interested in how a specific $x$ value behaves

$\hat{f} (x_{0}) = {\hat{β}}_{0} + {\hat{β}}_{1} x_{0} + {\hat{β}}_{2} x_{0}^{2} + {\hat{β}}_{3} x_{0}^{3} + {\hat{β}}_{4} x_{0}^{4}$

or more often a change between two values, $a$ and $b$

$\hat{f} (b) - \hat{f} (a) = {\hat{β}}_{1} b + {\hat{β}}_{2} b^{2} + {\hat{β}}_{3} b^{3} + {\hat{β}}_{4} b^{4} - {\hat{β}}_{1} a - {\hat{β}}_{2} a^{2} - {\hat{β}}_{3} a^{3} - {\hat{β}}_{4} a^{4}$

4 / 22

How is it done?

New variables are created ( $X_{1} = X$ , $X_{2} = X^{2}$ , $X_{3} = X^{3}$ , etc) and treated as multiple linear regression
We are not interested in the individual coefficients, we are interested in how a specific $x$ value behaves

$\hat{f} (x_{0}) = {\hat{β}}_{0} + {\hat{β}}_{1} x_{0} + {\hat{β}}_{2} x_{0}^{2} + {\hat{β}}_{3} x_{0}^{3} + {\hat{β}}_{4} x_{0}^{4}$

or more often a change between two values, $a$ and $b$

$\hat{f} (b) - \hat{f} (a) = {\hat{β}}_{1} (b - a) + {\hat{β}}_{2} (b^{2} - a^{2}) + {\hat{β}}_{3} (b^{3} - a^{3}) + {\hat{β}}_{4} (b^{4} - a^{4})$

4 / 22

Polynomial Regression

$\hat{f} (b) - \hat{f} (a) = {\hat{β}}_{1} (b - a) + {\hat{β}}_{2} (b^{2} - a^{2}) + {\hat{β}}_{3} (b^{3} - a^{3}) + {\hat{β}}_{4} (b^{4} - a^{4})$

How do you pick $a$ and $b$ ?

5 / 22

Polynomial Regression

$\hat{f} (b) - \hat{f} (a) = {\hat{β}}_{1} (b - a) + {\hat{β}}_{2} (b^{2} - a^{2}) + {\hat{β}}_{3} (b^{3} - a^{3}) + {\hat{β}}_{4} (b^{4} - a^{4})$

How do you pick $a$ and $b$ ?

If given no other information, a sensible choice may be the 25th and 75th percentiles of $x$

5 / 22

Polynomial Regression

6 / 22

03:00

`Polynomial Regression`

$p o p = β_{0} + β_{1} a g e + β_{2} a g e^{2} + β_{3} a g e^{3} + β_{4} a g e^{4} + ϵ$

Using the information below, write out the equation to predicted change in population from a change in age from the 25th percentile (24.5) to a 75th percentile (73.5).

term	estimate	std.error	statistic	p.value
(Intercept)	1672.0854	64.5606	25.8995	0.0000
age	-10.6429	9.2268	-1.1535	0.2516
I(age^2)	-1.1427	0.3857	-2.9627	0.0039
I(age^3)	0.0216	0.0059	3.6498	0.0004
I(age^4)	-0.0001	0.0000	-3.6540	0.0004

7 / 22

Choosing $d$

$y_{i} = β_{0} + β_{1} x_{i} + β_{2} x_{i}^{2} + β_{3} x_{i}^{3} \dots + β_{d} x_{i}^{d} + ϵ_{i}$

Either:

Pre-specify $d$ (before looking 👀 at your data!)
Use cross-validation to pick $d$

8 / 22

Choosing $d$

$y_{i} = β_{0} + β_{1} x_{i} + β_{2} x_{i}^{2} + β_{3} x_{i}^{3} \dots + β_{d} x_{i}^{d} + ϵ_{i}$

Either:

Pre-specify $d$ (before looking 👀 at your data!)
Use cross-validation to pick $d$

Why?

8 / 22

Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Polynomial Regressionpolynomials have notoriously bad tail behavior (so they can be bad for extrapolation)
9 / 22

Polynomial Regression

polynomials have notoriously bad tail behavior (so they can be bad for extrapolation)

What does this mean?

9 / 22

Step functions

Another way to create a transformation is to cut the variable into distinct regions

$C_{1} (X) = I (X < 35), C_{2} (X) = I (35 \leq X < 65), C_{3} (X) = I (X \geq 65)$

10 / 22

Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Step functionsCreate dummy variables for each group
11 / 22

Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Step functionsCreate dummy variables for each group
Include each of these variables in multiple regression
11 / 22

Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Step functionsCreate dummy variables for each group
Include each of these variables in multiple regression
The choice of cutpoints or knots can be problematic (and make a big difference!)
11 / 22

Step functions

$C_{1} (X) = I (X < 35), C_{2} (X) = I (35 \leq X < 65), C_{3} (X) = I (X \geq 65)$

12 / 22

Step functions

$C_{1} (X) = I (X < 35), C_{2} (X) = I (35 \leq X < 65), C_{3} (X) = I (X \geq 65)$

$C_{1} (X) = I (X < 15), C_{2} (X) = I (15 \leq X < 65), C_{3} (X) = I (X \geq 65)$

12 / 22

Piecewise polynomials

Instead of a single polynomial in $X$ over it's whole domain, we can use different polynomials in regions defined by knots

$y_{i} = {\begin{cases} β_{01} + β_{11} x_{i} + β_{21} x_{i}^{2} + β_{31} x_{i}^{3} + ϵ_{i} & if x_{i} < c \\ β_{02} + β_{12} x_{i} + β_{22} x_{i}^{2} + β_{32} x_{i}^{3} + ϵ_{i} & if x_{i} \geq c \end{cases}$

13 / 22

Piecewise polynomials

Instead of a single polynomial in $X$ over it's whole domain, we can use different polynomials in regions defined by knots

What could go wrong here?

13 / 22

Piecewise polynomials

Instead of a single polynomial in $X$ over it's whole domain, we can use different polynomials in regions defined by knots

What could go wrong here?

It would be nice to have constraints (like continuity!)

13 / 22

Piecewise polynomials

Instead of a single polynomial in $X$ over it's whole domain, we can use different polynomials in regions defined by knots

What could go wrong here?

It would be nice to have constraints (like continuity!)
Insert splines!

13 / 22

14 / 22

Linear Splines

A linear spline with knots at $ξ_{k}$ , $k = 1, \dots, K$ is a piecewise linear polynomial continuous at each knot

15 / 22

Linear Splines

A linear spline with knots at $ξ_{k}$ , $k = 1, \dots, K$ is a piecewise linear polynomial continuous at each knot

$y_{i} = β_{0} + β_{1} b_{1} (x_{i}) + β_{2} b_{2} (x_{i}) + \dots + β_{K + 1} b_{K + 1} (x_{i}) + ϵ_{i}$

15 / 22

Linear Splines

A linear spline with knots at $ξ_{k}$ , $k = 1, \dots, K$ is a piecewise linear polynomial continuous at each knot

$y_{i} = β_{0} + β_{1} b_{1} (x_{i}) + β_{2} b_{2} (x_{i}) + \dots + β_{K + 1} b_{K + 1} (x_{i}) + ϵ_{i}$

$b_{k}$ are basis functions

$\begin{aligned} b_{1} (x_{i}) & = x_{i} \\ b_{k + 1} (x_{i}) & = (x_{i} - ξ_{k})_{+}, k = 1, \dots, K \end{aligned}$

15 / 22

Linear Splines

A linear spline with knots at $ξ_{k}$ , $k = 1, \dots, K$ is a piecewise linear polynomial continuous at each knot

$y_{i} = β_{0} + β_{1} b_{1} (x_{i}) + β_{2} b_{2} (x_{i}) + \dots + β_{K + 1} b_{K + 1} (x_{i}) + ϵ_{i}$

$b_{k}$ are basis functions

$\begin{aligned} b_{1} (x_{i}) & = x_{i} \\ b_{k + 1} (x_{i}) & = (x_{i} - ξ_{k})_{+}, k = 1, \dots, K \end{aligned}$

Here $()_{+}$ means the positive part

$(x_{i} - ξ_{k})_{+} = {\begin{cases} x_{i} - ξ_{k} & if x_{i} > ξ_{k} \\ 0 & otherwise \end{cases}$

15 / 22

16 / 22

Cubic Splines

A cubic spliens with knots at $ξ_{i}, k = 1, \dots, K$ is a piecewise cubic polynomial with continuous derivatives up to order 2 at each knot.

17 / 22

Cubic Splines

A cubic spliens with knots at $ξ_{i}, k = 1, \dots, K$ is a piecewise cubic polynomial with continuous derivatives up to order 2 at each knot.

Again we can represent this model with truncated power functions

$y_{i} = β_{0} + β_{1} b_{1} (x_{i}) + β_{2} b_{2} (x_{i}) + \dots + β_{K + 3} b_{K + 3} (x_{i}) + ϵ_{i}$

$\begin{aligned} b_{1} (x_{i}) & = x_{i} \\ b_{2} (x_{i}) & = x_{i}^{2} \\ b_{3} (x_{i}) & = x_{i}^{3} \\ b_{k + 3} (x_{i}) & = (x_{i} - ξ_{k})_{+}^{3}, k = 1, \dots, K \end{aligned}$

17 / 22

Cubic Splines

A cubic spliens with knots at $ξ_{i}, k = 1, \dots, K$ is a piecewise cubic polynomial with continuous derivatives up to order 2 at each knot.

Again we can represent this model with truncated power functions

$y_{i} = β_{0} + β_{1} b_{1} (x_{i}) + β_{2} b_{2} (x_{i}) + \dots + β_{K + 3} b_{K + 3} (x_{i}) + ϵ_{i}$

$\begin{aligned} b_{1} (x_{i}) & = x_{i} \\ b_{2} (x_{i}) & = x_{i}^{2} \\ b_{3} (x_{i}) & = x_{i}^{3} \\ b_{k + 3} (x_{i}) & = (x_{i} - ξ_{k})_{+}^{3}, k = 1, \dots, K \end{aligned}$

where

$(x_{i} - ξ_{k})_{+}^{3} = {\begin{cases} (x_{i} - ξ_{k})^{3} & if x_{i} > ξ_{k} \\ 0 & otherwise \end{cases}$

17 / 22

18 / 22

Natural cubic splines

A natural cubic spline extrapolates linearly beyond the boundary knots

19 / 22

Natural cubic splines

A natural cubic spline extrapolates linearly beyond the boundary knots

This adds 4 extra constraints and allows us to put more internal knots for the same degrees of freedom as a regular cubic spline

19 / 22

Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Knot placementOne strategy is to decide KK (the number of knots) in advance and then place them at appropriate quantiles of the observed XX
20 / 22

Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Knot placementOne strategy is to decide KK (the number of knots) in advance and then place them at appropriate quantiles of the observed XX
A cubic spline with KK knots has K+4K+4 parameters (or degrees of freedom!)
20 / 22

Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

Knot placementOne strategy is to decide KK (the number of knots) in advance and then place them at appropriate quantiles of the observed XX
A cubic spline with KK knots has K+4K+4 parameters (or degrees of freedom!)
A natural spline with KK knots has KK degrees of freedom
20 / 22

Knot placement

Here is a comparision of a degree-14 polynomial and natural cubic spline (both have 15 degrees of freedom)

21 / 22

Dr. Lucy D'Agostino McGowan adapted from slides by Hastie & Tibshirani

22 / 22

↑, ←, Pg Up, k	Go to previous slide
↓, →, Pg Dn, Space, j	Go to next slide
Home	Go to first slide
End	Go to last slide
Number + Return	Go to specific slide
b / m / f	Toggle blackout / mirrored / fullscreen mode
c	Clone slideshow
p	Toggle presenter mode
t	Restart the presentation timer
?, h	Toggle this help

Non-linearity

Dr. D’Agostino McGowan

Non-linear relationships

Non-linear relationships

Non-linear relationships

Polynomials

Polynomials

Polynomials

How is it done?

How is it done?

How is it done?

How is it done?

Polynomial Regression

Polynomial Regression

Polynomial Regression

Polynomial Regression

Choosing dd

Either:

Choosing dd

Either:

Polynomial Regression

Polynomial Regression

Step functions

Step functions

Step functions

Step functions

Step functions

Step functions

Piecewise polynomials

Piecewise polynomials

Piecewise polynomials

Piecewise polynomials

Linear Splines

Linear Splines

Linear Splines

Linear Splines

Cubic Splines

Cubic Splines

Cubic Splines

Natural cubic splines

Natural cubic splines

Knot placement

Knot placement

Knot placement

Knot placement

Non-linear relationships

Help

`Polynomial Regression`

Choosing $d$

Choosing $d$