Git configuration

⊕Your email address is the address tied to your GitHub account and your name should be first and last name.

• Go to the Terminal pane
• Type the following two lines of code, replacing the information in the quotation marks with your info:

git config --global user.email "your email"
git config --global user.name "your name"

To confirm that the changes have been implemented, run the following

git config --global user.email
git config --global user.name

Password caching

If you would like your git password cached for a week for this project, type the following in the Terminal:

git config --global credential.helper 'cache --timeout 604800'

Project name:

Currently your project is called Untitled Project. Update the name of your project to be “Lab 03 - Cross Validation”.

YAML:

Open the R Markdown (Rmd) file in your project, change the author name to your name, and knit the document.

Commiting and pushing changes:

• Go to the Git pane in your RStudio.
• View the Diff and confirm that you are happy with the changes.
• Add a commit message like “Update team name” in the Commit message box and hit Commit.
• Click on Push. This will prompt a dialogue box where you first need to enter your user name, and then your password.

Conceptual questions

1. Explain how k-fold Cross Validation is implemented.

2. What are the advantages / disadvantages of k-fold Cross Validation compared to the Validation Set approach? What are the advantages / disadvantages of k-fold Cross Validation compared to Leave-one-out Cross Validation?

K-fold cross validation

We are trying to decide between three models of varying flexibility:

• Model 1: \(\texttt{mpg} = \beta_0 + \beta_1 \texttt{horsepower} + \epsilon\)
• Model 2: \(\texttt{mpg} = \beta_0 + \beta_1 \texttt{horsepower} + \beta_2 \texttt{horsepower}^2 + \epsilon\)
• Model 3: \(\texttt{mpg} = \beta_0 + \beta_1 \texttt{horsepower} + \beta_2 \texttt{horsepower}^2 + \beta_3 \texttt{horsepower}^3 + \epsilon\)

3. Using the Auto data, shuffle the rows and add a column, k, that groups the data into two groups, the first group, k = 1, will consist of 196 rows of the data to train the model, and k = 2 will consist of 196 rows to test the model. Make sure you set a seed so you end up with the same answer each time you run this.

set.seed(1)
Auto_2 <- Auto %>%
  --- %>% ## shuffle the data 
  --- ## add a column, k, to split the data into two groups

⊕You can use the poly() function to fit a model with a polynomial term. For example, to fit the model \(y = \beta_0 + \beta_1 \texttt{x} + \beta_2 \texttt{x}^2 + \beta_3 \texttt{x}^3 + \epsilon\), you would run lm(y ~ poly(x, 3), data = data)

4. Filter the data frame to the training data only. Fit the three models detailed above on this data. Then filter the data frame to the testing data only. Using the model created on the training data, predict mpg in the test data set. What is the test MSE for the three models? Which model would you choose?

5. Using the Auto data, shuffle the rows and add a column, k, that groups the data into five groups of approximately equal size. Make sure you set a seed so you end up with the same answer each time you run this.

6. Create a function that generalizes the approach in Exercises 3 and 4, with K as a parameter that can vary. The function should output a data frame that contains four columns:

• MSE_1 - the mean squared error for Model 1
• MSE_2 - the mean squared error for Model 2
• MSE_3 - the mean squared error for Model 3
• n_k - the sample size for the fold

7. Map this function from Exercise 6 over the 5 folds and calculate the overall 5-fold Cross Validation error for each of the three models. Which model would you chose?