Study Sessions

• Monday 7-9p
• Manchester 122

Lab 01

• Knit, Commit, Push often
• Commit and Push all files
• Check on GitHub.com to make sure everything is updating
• You won't see a rendered file

Regression and Classification

• Regression: quantitative response
• Classification: qualitative (categorical) response

Regression and Classification

• Regression: quantitative response
• Classification: qualitative (categorical) response

Regression and Classification

• Regression: quantitative response
• Classification: qualitative (categorical) response

Auto data

Above are mpg vs horsepower, weight, and acceleration, with a blue linear-regression line fit separately to each. Can we predict mpg using these three?

Maybe we can do better using a model:

$\text{mpg}\approx f(\text{horsepower},\text{weight},\text{acceleration})$

Notation

• mpg is the response variable, the outcome variable, we refer to this as $Y$
• horsepower is a feature, input, predictor, we refer to this as $X_1$
• weight is $X_2$
• acceleration is $X_3$ Our *input vector is

$$X=\left[\begin{array}{c}{X}_1\\ X_2\\ X_3\end{array}\right]$$

• Our model is

$$Y=f\left(X\right)+ϵ$$

• $ϵ$ is our error

Why do we care about $f\left(X\right)$?

• We can use $f\left(X\right)$ to make predictions of $Y$ for new values of $X=x$ We can gain a better understanding of which components of $X=(X_1,X_2,\dots ,X_p)$ are important for explaining $Y$ Depending on how complex $f$ is, maybe we can understand how each component ( $X_j$ ) of $X$ affects $Y$

How do we choose $f\left(X\right)$? What is a good value for $f\left(X\right)$ at any selected value of $X$, say $X=100$? There can be many $Y$ values at $X=100$. A good value is

$$f\left(100\right)=E(Y|X=100)$$

$E(Y|X=100)$ means expected value (average) of $Y$ given $X=100$

This ideal $f\left(x\right)=E(Y|X=x)$ is called the regression function

Regression function, $f\left(X\right)$

• Also works or a vector, $X$, for example,

$$f\left(x\right)=f(x_1,x_2,x_3)=E[Y|X_1=x_1,X_2=x_2,X_3=x_3]$$

• This is the optimal predictor of $Y$ in terms of mean-squared prediction error

  $f\left(x\right)=E(Y|X=x)$ is the function that minimizes $E\left[\right(Y-g\left(X\right){)}^2|X=x]$ over all functions $g$ at all points $X=x$

• $ϵ=Y-f\left(x\right)$ is the irreducible error
• even if we knew $f\left(x\right)$, we would still make errors in prediction, since at each $X=x$ there is typically a distribution of possible $Y$ values

• Take the average! $f\left(100\right)=E[\text{mpg}|\text{horsepower}=100]=19.6$

• $ϵ=Y-f\left(X\right)=32.9-19.6=13.3$

The error

For any estimate, $\hat{f}\left(x\right)$, of $f\left(x\right)$, we have

$$E\left[\right(Y-\hat{f}\left(x\right){)}^2|X=x]=\underset{\text{reducible error}}{\underset{}{\left[f\right(x)-\hat{f}(x){]}^2}}+\underset{\text{irreducible error}}{\underset{}{Var\left(ϵ\right)}}$$

Estimating $f$

• Typically we have very few (if any!) data points at $X=x$ exactly, so we cannot compute $E[Y|X=x]$* For example, what if we were interested in estimating miles per gallon when horsepower was 104.

💡 We can relax the definition and let

$$\hat{f}\left(x\right)=E[Y|X\in N(x\left)\right]$$

Estimating $f$

• Typically we have very few (if any!) data points at $X=x$ exactly, so we cannot compute $E[Y|X=x]$
• For example, what if we were interested in estimating miles per gallon when horsepower was 104.

💡 We can relax the definition and let

$$\hat{f}\left(x\right)=E[Y|X\in N(x\left)\right]$$

• Where $N\left(x\right)$ is some neighborhood of $x$

Notation pause!

$$\hat{f}\left(x\right)=\underset{\text{The expectation}}{\underset{}{E}}\left[\underset{\text{of Y}}{\underset{}{Y}}\underset{\text{given}}{\underset{}{|}}\underset{\text{X is in the neighborhood of x}}{\underset{}{X\in N\left(x\right)}}\right]$$

Estimating $f$

💡 We can relax the definition and let

$$\hat{f}\left(x\right)=E[Y|X\in N(x\left)\right]$$

• Nearest neighbor averaging does pretty well with small $p$ ( $p\le 4$ ) and large $n$ Nearest neighbor is not great when $p$ is large because of the *curse of dimensionality (because nearest neighbors tend to be far away in high dimensions)

  What do I mean by $p$? What do I mean by $n$?

Parametric models

A common parametric model is a linear model

$$f\left(X\right)={\beta }_0+{\beta }_1{X}_1+{\beta }_2{X}_2+\cdots +{\beta }_p{X}_p$$

• A linear model has $p+1$ parameters ( ${\beta }_0,\dots ,{\beta }_p$ ) We estimate these parameters by fitting a model to training data Although this model is almost never correct it can often be a good interpretable approximation to the unknown true function, $f\left(X\right)$

• The red points are simulated values for income from the model:

$$\text{income}=f\left(\text{education, senority}\right)+ϵ$$

• $f$ is the blue surface

Linear regression model fit to the simulated data

$${\hat{f}}_L\left(\text{education, senority}\right)={\hat{\beta }}_0+{\hat{\beta }}_1\text{education}+{\hat{\beta }}_2\text{senority}$$

• More flexible regression model ${\hat{f}}_S\left(\text{education, seniority}\right)$ fit to the simulated data
• Here we use a technique called a thin-plate spline to fit a flexible surface

And even MORE flexible 😱 model $\hat{f}\left(\text{education, seniority}\right)$

• Here we've basically drawn the surface to hit every point, minimizing the error, but completely overfitting

🤹 Finding balance

• Prediction accuracy versus interpretability
• Linear models are easy to interpret, thin-plate splines are not Good fit versus overfit or *underfit
• How do we know when the fit is just right? Parsimony versus *black-box
• We often prefer a simpler model involving fewer variables over a black-box predictor involving them all

Accuracy

• We've fit a model $\hat{f}\left(x\right)$ to some training data $\text{train}=\{x_i,y_i{\}}_1^N$
• We can measure accuracy as the average squared prediction error over that train data

$$MS{E}_{\text{train}}={\text{Ave}}_{i\in \text{train}}[y_i-\hat{f}(x_i){]}^2$$

• This may be biased towards overfit models

Accuracy

$$MS{E}_{\text{train}}={\text{Ave}}_{i\in \text{train}}[y_i-\hat{f}(x_i){]}^2$$

Accuracy

$$MS{E}_{\text{train}}={\text{Ave}}_{i\in \text{train}}[y_i-\hat{f}(x_i){]}^2$$

Accuracy

It's overfit!

Accuracy

If we get a new sample, that overfit model is probably going to be terrible!

Accuracy

• We've fit a model $\hat{f}\left(x\right)$ to some training data $\text{train}=\{x_i,y_i{\}}_1^N$
• Instead of measuring accuracy as the average squared prediction error over that train data, we can compute it using fresh test data $\text{test}=\{x_i,y_i{\}}_1^M$

$$MS{E}_{\text{test}}={\text{Ave}}_{i\in \text{test}}[y_i-\hat{f}(x_i){]}^2$$

Black curve is the "truth" on the left. Red curve on right is $MS{E}_{\text{test}}$, grey curve is $MS{E}_{\text{train}}$. Orange, blue and green curves/squares correspond to fis of different flexibility.

Here the truth is smoother, so the smoother fit and linear model do really well

Here the truth is wiggly and the noise is low, so the more flexible fits do the best

Bias-variance trade-off

• We've fit a model, $\hat{f}\left(x\right)$, to some training data Let's pull a test observation from this population ( $x_0,y_0$ ) The true model is $Y=f\left(x\right)+ϵ$* $f\left(x\right)=E[Y|X=x]$

$$E(y_0-\hat{f}(x_0){)}^2=\text{Var}(\hat{f}\left(x_0\right))+[\text{Bias}\left(\hat{f}\right(x_0\left)\right){]}^2+\text{Var}\left(ϵ\right)$$

The expectation averages over the variability of $y_0$ as well as the variability of the training data. $\text{Bias}\left(\hat{f}\right(x_0\left)\right)=E\left[\hat{f}\right(x_0\left)\right]-f\left(x_0\right)$

• As flexibility of $\hat{f}$ $\uparrow$, its variance $\uparrow$ and its bias $\downarrow$ choosing the flexibility based on average test error amounts to a *bias-variance trade-off

Bias-variance trade-off

Notation

• $Y$ is the response variable. It is qualitative
• $C\left(X\right)$ is the classifier that assigns a class $C$ to some future unlabeled observation, $X$* Examples:
  • Email can be classified as $C=\left(\text{spam, not spam}\right)$
  • Written number is one of $C=\{0,1,2,\dots ,9\}$

Classification Problem

• Build a classifier $C\left(X\right)$ that assigns a class label from $C$ to a future unlabeled observation $X$
• Assess the uncertainty in each classification
• Understand the roles of the different predictors among $X=(X_1,X_2,\dots ,X_p)$

Suppose there are $K$ elements in $C$, numbered $1,2,\dots ,K$

$$p_k\left(x\right)=P(Y=k|X=x),k=1,2,\dots ,K$$ These are conditional class probabilities at $x$

• In the plot, you could examine the mini-barplot at $x=5$

Suppose there are $K$ elements in $C$, numbered $1,2,\dots ,K$

$$p_k\left(x\right)=P(Y=k|X=x),k=1,2,\dots ,K$$ These are conditional class probabilities at $x$

• The Bayes optimal classifier at $x$ is

$$C\left(x\right)=j\text{if}{p}_j\left(x\right)=\text{max}\left\{p_1\right(x),p_2(x),\dots ,p_K(x\left)\right\}$$

• Nearest neighbor like before!* This does break down as the dimensions grow, but the impact of $\hat{C}\left(x\right)$ is less than on ${\hat{p}}_k\left(x\right),k=1,2,\dots ,K$

Accuracy

• Misclassification error rate

$$Er{r}_{\text{test}}={\text{Ave}}_{i\in \text{test}}I[y_i\ne \hat{C}(x_i\left)\right]$$

• The Bayes Classifier using the true $p_k\left(x\right)$ has the smallest error Some of the methods we will learn build structured models for $C\left(x\right)$ (support vector machines, for example) Some build structured models for $p_k\left(x\right)$ (logistic regression, for example)

K-Nearest-Neighbors example

KNN (K = 10)

KNN

Trade-offs