Lab 3 — Correlation, Regression & Residuals

Quantitative Reasoning · LAB 413

How to use this document. Run the code along with me. Comments start with #. Each section mirrors the live lab; code chunks show the command and its output together.

Today’s plan

1. Clean environment + setup
2. Import & understand School data
3. Make a smaller data frame (base vs tidyverse)
4. Descriptive stats (mean, sd)
5. Correlation
6. Linear regression
7. Residuals (what they are & how to plot)
8. Quick practice

---

1 Clean Environment + Setup

Always follow these steps when you open RStudio:

1. Install packages (only once on your computer)
2. Load packages (every new session)
3. Set your working directory
4. Import your dataset

rm(list = ls())   # clean previous objects so we start fresh

# Load packages (every new R session)
library(tidyverse)   # collection of core data science packages (ggplot2, readr, ...)
library(dplyr)       # data manipulation (select, filter, mutate, summarize)
library(ggplot2)     # data visualization (plots, graphs)
library(conflicted)  # handles conflicts when two packages share function names

# If function names conflict (e.g., filter from dplyr vs stats), prefer dplyr's:
conflicts_prefer(dplyr::filter)

In the live session we set the working directory with setwd(...). In this document the data file SchoolData.csv sits next to the .Rmd, so no setwd() is needed.

2 Import & Understand School Data

Data source: School Discipline & Attendance (Philadelphia).

schooldata <- read.csv("SchoolData.csv")

# In RStudio you would run View(schooldata); here we preview the first rows.
knitr::kable(head(schooldata))

year_academic	id_school	name_school	sector	category	group	oss_denom	oss_0_numer	oss_0_pct	oss_1_numer	oss_1_pct	oss_2_numer	oss_2_pct	oss_3_numer	oss_3_pct	oss_4_numer	oss_4_pct	att_denom	att_95_numer	att_95_pct	grad_denom	grad_4_numer	grad_4_pct
2018-2019	1010	John Bartram High School	District	All Students	All Students	683	579	84.77	84	12.30	14	2.05	6	0.88	0	0.00	683	174	25.48	159	94	59.12
2018-2019	1020	West Philadelphia High School	District	All Students	All Students	501	421	84.03	61	12.18	12	2.40	4	0.80	3	0.60	501	68	13.57	114	70	61.40
2018-2019	1030	High School of the Future	District	All Students	All Students	561	507	90.37	40	7.13	12	2.14	1	0.18	1	0.18	561	251	44.74	107	88	82.24
2018-2019	1050	Paul Robeson High School for Human Services	District	All Students	All Students	323	308	95.36	14	4.33	1	0.31	0	0.00	0	0.00	323	122	37.77	76	71	93.42
2018-2019	1100	William L. Sayre High School	District	All Students	All Students	539	418	77.55	83	15.40	30	5.57	6	1.11	2	0.37	539	84	15.58	103	55	53.40
2018-2019	1130	William T. Tilden School	District	All Students	All Students	457	362	79.21	59	12.91	16	3.50	9	1.97	11	2.41	457	233	50.98	NA	NA	NA

Codebook

• year_academic = school year
• id_school = school id
• name_school = school name
• oss_denom = total # of students (suspension denominator)
• oss_0_numer = # with 0 suspensions
• oss_0_pct = % with 0 suspensions
• att_denom = total # for attendance
• att_95_numer = # who attended 95%+ days
• att_95_pct = % who attended 95%+ days

“Out-of-school suspension” (OSS) = kept out of school up to 10 days.

# Quick peek
dim(schooldata)

#> [1] 213  23

head(schooldata)

str(schooldata)

#> 'data.frame':    213 obs. of  23 variables:
#>  $ year_academic: chr  "2018-2019" "2018-2019" "2018-2019" "2018-2019" ...
#>  $ id_school    : int  1010 1020 1030 1050 1100 1130 1190 1200 1230 1250 ...
#>  $ name_school  : chr  "John Bartram High School" "West Philadelphia High School" "High School of the Future" "Paul Robeson High School for Human Services" ...
#>  $ sector       : chr  "District" "District" "District" "District" ...
#>  $ category     : chr  "All Students" "All Students" "All Students" "All Students" ...
#>  $ group        : chr  "All Students" "All Students" "All Students" "All Students" ...
#>  $ oss_denom    : int  683 501 561 323 539 457 417 764 525 536 ...
#>  $ oss_0_numer  : int  579 421 507 308 418 362 383 722 483 534 ...
#>  $ oss_0_pct    : num  84.8 84 90.4 95.4 77.5 ...
#>  $ oss_1_numer  : int  84 61 40 14 83 59 31 35 29 1 ...
#>  $ oss_1_pct    : num  12.3 12.18 7.13 4.33 15.4 ...
#>  $ oss_2_numer  : int  14 12 12 1 30 16 3 5 13 1 ...
#>  $ oss_2_pct    : num  2.05 2.4 2.14 0.31 5.57 3.5 0.72 0.65 2.48 0.19 ...
#>  $ oss_3_numer  : int  6 4 1 0 6 9 0 2 0 0 ...
#>  $ oss_3_pct    : num  0.88 0.8 0.18 0 1.11 1.97 0 0.26 0 0 ...
#>  $ oss_4_numer  : int  0 3 1 0 2 11 0 0 0 0 ...
#>  $ oss_4_pct    : num  0 0.6 0.18 0 0.37 2.41 0 0 0 0 ...
#>  $ att_denom    : int  683 501 561 323 539 457 417 764 525 536 ...
#>  $ att_95_numer : int  174 68 251 122 84 233 327 211 119 246 ...
#>  $ att_95_pct   : num  25.5 13.6 44.7 37.8 15.6 ...
#>  $ grad_denom   : int  159 114 107 76 103 NA 88 NA NA NA ...
#>  $ grad_4_numer : int  94 70 88 71 55 NA 75 NA NA NA ...
#>  $ grad_4_pct   : num  59.1 61.4 82.2 93.4 53.4 ...

3 Make a Smaller Data Frame

We keep only the columns we need. Here are two equivalent ways.

# Base R way (subset + select)
df1 <- subset(schooldata,
              select = c(year_academic, id_school, name_school,
                         oss_denom, oss_0_numer, oss_0_pct,
                         att_denom, att_95_numer, att_95_pct)
)

# tidyverse way (dplyr::select + pipe)
df <- schooldata %>%
  select(year_academic, id_school, name_school,
         oss_denom, oss_0_numer, oss_0_pct,
         att_denom, att_95_numer, att_95_pct)

# Check
knitr::kable(head(df1))

year_academic	id_school	name_school	oss_denom	oss_0_numer	oss_0_pct	att_denom	att_95_numer	att_95_pct
2018-2019	1010	John Bartram High School	683	579	84.77	683	174	25.48
2018-2019	1020	West Philadelphia High School	501	421	84.03	501	68	13.57
2018-2019	1030	High School of the Future	561	507	90.37	561	251	44.74
2018-2019	1050	Paul Robeson High School for Human Services	323	308	95.36	323	122	37.77
2018-2019	1100	William L. Sayre High School	539	418	77.55	539	84	15.58
2018-2019	1130	William T. Tilden School	457	362	79.21	457	233	50.98

knitr::kable(head(df))

year_academic	id_school	name_school	oss_denom	oss_0_numer	oss_0_pct	att_denom	att_95_numer	att_95_pct
2018-2019	1010	John Bartram High School	683	579	84.77	683	174	25.48
2018-2019	1020	West Philadelphia High School	501	421	84.03	501	68	13.57
2018-2019	1030	High School of the Future	561	507	90.37	561	251	44.74
2018-2019	1050	Paul Robeson High School for Human Services	323	308	95.36	323	122	37.77
2018-2019	1100	William L. Sayre High School	539	418	77.55	539	84	15.58
2018-2019	1130	William T. Tilden School	457	362	79.21	457	233	50.98

names(df1)

#> [1] "year_academic" "id_school"     "name_school"   "oss_denom"    
#> [5] "oss_0_numer"   "oss_0_pct"     "att_denom"     "att_95_numer" 
#> [9] "att_95_pct"

names(df)

#> [1] "year_academic" "id_school"     "name_school"   "oss_denom"    
#> [5] "oss_0_numer"   "oss_0_pct"     "att_denom"     "att_95_numer" 
#> [9] "att_95_pct"

4 Descriptive Stats (mean, sd)

# Mean & SD for x = oss_0_pct (percent with zero suspensions)
mean(df$oss_0_pct)

#> [1] 93.63122

sd(df$oss_0_pct, na.rm = TRUE)   # na.rm = TRUE ignores missing values (NA)

#> [1] 6.386506

# Mean & SD for y = att_95_pct (percent with 95%+ attendance)
mean(df$att_95_pct, na.rm = TRUE)

#> [1] 45.66742

sd(df$att_95_pct, na.rm = TRUE)

#> [1] 16.85575

TODO. Compute min/max for oss_0_pct (% with 0 suspensions) and att_95_pct (% who attended 95%+ days).

5 Correlation

# Correlation r between oss_0_pct (x) and att_95_pct (y)
cor(df$oss_0_pct, df$att_95_pct, use = "complete.obs")

#> [1] 0.5522232

# use = "complete.obs": ignore missing values (NA) when computing

Interpretation. r close to 1 means a strong positive relationship; close to -1 means a strong negative one; around 0 means little or no (linear) relationship.

TODO. Compute the correlation between oss_0_numer (# with 0 suspensions) and att_95_numer (# who attended 95%+ days).

6 Linear Regression

Regression is about predicting a relationship: drawing the best-fit line through the data. To estimate the regression we use lm(y ~ x, data = ).

# Model: predict attendance % (y) from % zero suspensions (x)
m1 <- lm(att_95_pct ~ oss_0_pct, data = df)

m1          # quick equation

#> 
#> Call:
#> lm(formula = att_95_pct ~ oss_0_pct, data = df)
#> 
#> Coefficients:
#> (Intercept)    oss_0_pct  
#>     -90.797        1.457

summary(m1) # details

#> 
#> Call:
#> lm(formula = att_95_pct ~ oss_0_pct, data = df)
#> 
#> Residuals:
#>     Min      1Q  Median      3Q     Max 
#> -27.171 -10.417  -1.643   7.405  35.349 
#> 
#> Coefficients:
#>             Estimate Std. Error t value Pr(>|t|)    
#> (Intercept) -90.7972    14.2159  -6.387 1.06e-09 ***
#> oss_0_pct     1.4575     0.1515   9.622  < 2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Residual standard error: 14.09 on 211 degrees of freedom
#> Multiple R-squared:  0.305,  Adjusted R-squared:  0.3017 
#> F-statistic: 92.58 on 1 and 211 DF,  p-value: < 2.2e-16

• Intercept = predicted att_95_pct when oss_0_pct = 0 (not always meaningful)
• Slope = how much att_95_pct changes for a 1-point increase in oss_0_pct, on average
• e: “times 10 to the power of” (e.g., 1.06e-09 = 1.06 * 10^{-9} = 0.00000000106)
• R-squared ≈ % of variance in y explained by x

# Scatterplot + regression line
ggplot(df, aes(x = att_95_pct, y = oss_0_pct)) +
  geom_point() +
  labs(title = "scatterplot",
       x = "% who attended 95%+ days",
       y = "% with 0 suspensions") +
  geom_smooth(method = lm, color = "red") # regression line

# gray shaded area: confidence interval (the margin of uncertainty)

TODO (Regression practice).

• Run a regression with att_95_numer (Y) ~ oss_0_numer (X).
• Then draw a scatterplot with a regression line using ggplot2.

7 Residuals (what & how to plot)

Residuals = actual − predicted.

df$Residuals <- residuals(m1)

# Residual plot (x vs residuals) — should be centered around 0 with no pattern
ggplot(df, aes(x = oss_0_pct, y = Residuals)) +
  geom_point() +
  geom_hline(yintercept = 0, color = "darkred")

8 Quick Practice

P1) Make a smaller data set for ONE school year only (e.g., 2018–2019). Hint: use dplyr::filter.

P2) Using this smaller dataset, run a regression att_95_numer ~ oss_0_numer and draw a scatterplot with a regression line using ggplot2.

9 Wrap-up

Key takeaways

• A clean start helps: rm(list = ls()), load packages, setwd(), import.
• Subset the exact columns you need (base or tidyverse).
• Check r for strength/direction; then fit lm() for slope/intercept.
• Residuals = errors → look for “no pattern” around 0.