--- title: "Cross Validation" author: "DSC365: Introduction to Data Science" date: "October 20, 2026" format: revealjs: echo: true toc: true toc-depth: 1 toc-title: Outline slide-number: true menu: numbers: true --- ```{r echo=FALSE, message=FALSE, warning = FALSE} library(tidyverse) library(knitr) library(RColorBrewer) library(mosaic) library(infer) hook_output = knit_hooks$get('output') knit_hooks$set(output = function(x, options) { # this hook is used only when the linewidth option is not NULL if (!is.null(n <- options$linewidth)) { x = xfun::split_lines(x) # any lines wider than n should be wrapped if (any(nchar(x) > n)) x = strwrap(x, width = n) x = paste(x, collapse = '\n') } hook_output(x, options) }) ``` # Announcements {.unnumbered .unlisted} **Mini Project 2** - Due _this_ Thursday, October 22, 11:59pm in Blueline **Lab 5** (Linear Models) - Due _next_ Tuesday October 27 at 11:59 pm in Blueline **Quiz 3**: in class next Thursday October 29 - Covers: Prediction/KNN/Tree-Based Methods [Link to source code for these slides](cv.qmd) # Branches of Machine Learning {.smaller} **Machine Learning**: a broad term, coined in the 1950s, describing a set of inter-related techniques for extracting info from data **Supervised Learning**: Modeling a specific response from "labeled" data. That is, we have a set of input-output pairs. **Unsupervised Learning**: Approach to finding patterns where there is no clear response variable. # Models ## Prediction There are two things we can do with fitting a model: 1. Interpretation 2. Prediction Calculating a prediction is easy: Plug in your predictor variables into the formula defining the model to get the predicted response $\hat{y}$ Getting a good prediction is hard: ## No one model is "best" _"All models are wrong, but some are useful"_ - Prediction Accuracy and Model Interpretability Trade-Off + simple models are easy to interpret (e.g. coefficients of a linear regression), but might not be accurate for more complex data. + complicated models can be more accurate, but no one knows what any particular parameter does. - Bias vs Variance Trade-Off + we generally want low bias, low variance. + But mathematically, these are often mutually incompatible goals. ## Bias vs Variance ```{r echo=FALSE, fig.align='center', out.width="70%", fig.alt = "Targets comparing bias and variance. Low variance and high bias has observations that are off target but are close together. High variance and high bias observations are also off target, but are more spread out. Low variance and low bias have observations that are on target with little variability. High variance and low bias has observations that are generally oon target but has a lot of variability."} knitr::include_graphics("images/bias-variance.png") ``` ## Over vs Under Fitting ```{r echo=FALSE, fig.align='center', fig.alt="A plot to show the difference between over and underfitting. Over fitted curve follows the noise and is jagged as it follows the observations to closely. While the underfitted curve is just a line and doesn't capture the curvature in the data at all. A balance between these is necessary."} knitr::include_graphics("images/model-fit.png") ``` # Splitting Data ## Test vs. Train - Several steps to create a useful statistical model: parameter estimation, model selection, performance assessment, etc. - Doing all of this on the entire data we have available can lead to overfitting - To avoid overfitting, we split the data. + split the data into TEST and TRAIN + That is, we fit the model to the training data (TRAIN) and then measure the "quality" of the model on the testing data (TEST) ## Example: Flight Data Consider a random sample of 1000 flights from NYC to Chicago in 2013. We want to create a model to predict arrival delay. ```{r, message=FALSE} library(tidyverse) library(nycflights13) set.seed(14) Chicago1000 <- flights %>% filter(dest %in% c('ORD', 'MDW'), !is.na(arr_delay)) %>% sample_n(size=1000) ``` + Split it into a training and a testing set. ```{r, echo=TRUE} set.seed(365) test_id <- sample(1:nrow(Chicago1000), size=round(0.2*nrow(Chicago1000))) TEST <- Chicago1000[test_id,] TRAIN <- Chicago1000[-test_id,] ``` ## Example: Flight Data {.unlisted} + Fit model to Training set: ```{r} model1 = lm(arr_delay ~ hour + dep_delay, data = TRAIN) ``` This finds the coefficients in the model $y = \beta_0 + \beta_{hour} X_{hour} + \beta_{dep delay} X_{dep delay}$ + Predict outcome on the Testing Set: ```{r} predictions <- predict(model1, TEST) head(predictions) ``` Plugs all of the TEST values of `hour` and `dep_delay` into the model above # Model evaluation ## Is that right? We can compare the predicted values to the actual values: ```{r} head(TEST$dep_delay) # actual value head(predictions) # predicted value ``` ## Evaluate Performance: RMSE {.scrollable} Root Mean Square Error (RMSE) - for numerical response $$ \text{RMSE} = \sqrt{\frac{\sum^{n}_{i=1}(y_i - \hat{y}_i)^2}{n}} $$ RMSE for Test Set: ```{r, message=FALSE} library(Metrics) rmse(TEST$arr_delay, predictions) ``` Penalizes large errors more heavily (because error is squared), so it's sensitive to outliers. To give the number meaning, compare it to, say, the standard deviation in the arrival delays ```{r} sd(Chicago1000$arr_delay) ``` The RMSE (15.8) is smaller than the standard deviation of the response variable (47.1), which is a good sign ## Evaluate Performance: MAE Mean Absolute Error (MAE) - for numerical response $$\text{MAE} = \frac{1}{n} \sum^{n}_{i=1}|y_i - \hat{y}_i|$$ MAE for Test Set: ```{r} mae(TEST$arr_delay, predictions) ``` ## Model Comparison ```{r, echo = FALSE} options(pillar.sigfig = 5) ``` ```{r, echo=TRUE} model2 = lm(arr_delay ~ hour + dep_delay + distance, data = TRAIN) predictions_distance <- predict(model2, TEST) TEST %>% summarize( MSE1 = rmse(arr_delay, predictions), MSE2 = rmse(arr_delay, predictions_distance)) TEST %>% summarize( MAE1 = mae(arr_delay, predictions), MAE2 = mae(arr_delay, predictions_distance)) ``` # Cross Validation ## Problems with single splits {.scrollable} Potential Problem with a single split into training/testing: you evaluated the model only once and you are not sure your good result is by luck or not - when the data set is small, the chances we were lucky is higher (i.e. for very large data sets, there's usually less of a need for extensive cross-validation) - Basically, we want to evaluate the model multiple times to be confident in the fit. We can easily do this using cross validation - A resampling method that uses different portions of the data to test and train a model on different iterations ## $k$-fold Cross Validation {.smaller} - Split the dataset into $k$ subsets randomly + Generally choose $k = 5$ or $k = 10$. - Use k-1 subsets for training the model - Test the model against that one subset that was left in the previous step ```{r echo=FALSE, fig.align='center', out.width="100%", fig.alt = "Visual depiction of 5-fold cross validation with no validation set. The data is split into five sections where eachs section becomes the testing set one time resulting in 5 iterations. The average of the error for each of these iterations is calulcated and returned."} knitr::include_graphics("images/k-fold-cv1.png") ``` ## $k$-fold Cross Validation in `R` ```{r, message=FALSE} library(caret) train_control <- trainControl(method = "cv", number = 5) model <- train(arr_delay ~ hour + dep_delay, data = Chicago1000, trControl = train_control, method = "lm") model ``` ## Dealing with Hyperparameters If your model has hyperparameters, best to do it this way: ```{r echo=FALSE, fig.align='center', out.width="80%", fig.alt = "Visual depiction of 5-fold cross validation with a validation set. The data is split into five sections where eachs section becomes the validation set one time resulting in 5 iterations. This is used to help tune the parameters which are then used on the testing set."} knitr::include_graphics("images/k-fold-cv2.png") ``` Withhold a portion of the data for testing, and then do cross-validation within the testing set Note: Linear regression does not have hyperparameters, but we will discuss methods that do later in the course. # Caveats ## Caution with Linear Models: Extrapolation We extrapolate when we use the regression equation to produce a response value from an x-value that is outside the range of the observed x-values **Example:** Relationship between year and diameter of a dinner plate. ```{r echo=FALSE, fig.align='center', out.width="100%", fig.alt="Scatterplot depicting the relationship between year and plate size, which is moderatly strong and positive."} knitr::include_graphics("images/extrapolation.png") ```