Lab 2 — Visualization, Transformation, Correlation & Regression

Quantitative Reasoning · LAB 413

How to use this document. Run the code along with me. Comments start with #. Each section mirrors the live lab; code chunks show the command and its output together.

Today’s plan

1. Quick reminders: new R session setup
2. Working with data frames (review from Lab 1)
3. Histograms and boxplots (Base R)
4. Visualization with ggplot2
5. Variable transformation
6. Correlation
7. Linear regression

---

1 Quick Reminders: New R Session Setup

Always follow these steps when you open RStudio:

1. Install packages (only once on your computer).
2. Load packages (every new session).
3. Set your working directory.
4. Import your dataset.

# install.packages("tidyverse")   # only once
# install.packages("dplyr")

# Load packages
library(tidyverse)
library(dplyr)

In the live session we set the working directory with setwd(...). In this document the data files sit next to the .Rmd, so no setwd() is needed.

2 Working with Data Frames

2.1 Loading a Dataset

.rda files are R’s native format (rda stands for RData). Read them with load(), which puts the dataset into your environment under its original name.

load("EG01-19COLLEGE.rda")

# Best practice: rename it to a simpler name you can use easily.
# newname <- `oldname`  (wrap the original name in backticks)
mydata <- `EG01-19COLLEGE`

.csv files are spreadsheets saved as plain text with comma-separated values — the most common data format in data science. Read them with read.csv().

df <- read.csv("EG01-19COLLEGE.csv", header = TRUE)
# header = TRUE: treat the first row as column names instead of data values

A simple way to load data in RStudio: File → Open File → click the .RData or .csv file.

# Remove datasets we no longer need from memory
rm(`EG01-19COLLEGE`)
rm(mydata)

2.2 Explore the Data Frame

In RStudio you would run View(df); here we preview the first rows instead.

knitr::kable(head(df), caption = "First rows of the College data")

First rows of the College data

State	Undergrads	Population	UGradPerThou
New Jersey	326358	8640218	37.77196
Nevada	100760	2484196	40.56041
Alaska	27463	676301	40.60766
Georgia	378947	9318715	40.66516
Connecticut	142926	3487896	40.97771
Tennessee	250974	6068306	41.35816

str(df)        # structure: types + sample values

#> 'data.frame':    50 obs. of  4 variables:
#>  $ State       : chr  "New Jersey" "Nevada" "Alaska" "Georgia" ...
#>  $ Undergrads  : int  326358 100760 27463 378947 142926 250974 775171 187254 58512 57527 ...
#>  $ Population  : int  8640218 2484196 676301 9318715 3487896 6068306 18019093 4324799 1313355 1275264 ...
#>  $ UGradPerThou: num  37.8 40.6 40.6 40.7 41 ...

head(df, 10)   # first ten rows

df$State       # access a column by name

#>  [1] "New Jersey"     "Nevada"         "Alaska"         "Georgia"       
#>  [5] "Connecticut"    "Tennessee"      "Florida"        "South Carolina"
#>  [9] "Maine"          "Hawaii"         "New Hampshire"  "Montana"       
#> [13] "Maryland"       "Louisiana"      "Oregon"         "Mississippi"   
#> [17] "Ohio"           "Arkansas"       "Pennsylvania"   "Texas"         
#> [21] "New York"       "Alabama"        "West Virginia"  "Idaho"         
#> [25] "North Carolina" "Washington"     "Delaware"       "Indiana"       
#> [29] "Virginia"       "Oklahoma"       "Massachusetts"  "Kentucky"      
#> [33] "Missouri"       "Colorado"       "Illinois"       "Wisconsin"     
#> [37] "Michigan"       "South Dakota"   "Minnesota"      "Vermont"       
#> [41] "New Mexico"     "Nebraska"       "California"     "Wyoming"       
#> [45] "Kansas"         "Rhode Island"   "North Dakota"   "Utah"          
#> [49] "Iowa"           "Arizona"

dim(df)        # rows x columns

#> [1] 50  4

nrow(df)       # number of rows

#> [1] 50

ncol(df)       # number of columns

#> [1] 4

names(df)      # variable names

#> [1] "State"        "Undergrads"   "Population"   "UGradPerThou"

colnames(df)   # same as names()

#> [1] "State"        "Undergrads"   "Population"   "UGradPerThou"

rownames(df)   # row names (rarely needed)

#>  [1] "1"  "2"  "3"  "4"  "5"  "6"  "7"  "8"  "9"  "10" "11" "12" "13" "14" "15"
#> [16] "16" "17" "18" "19" "20" "21" "22" "23" "24" "25" "26" "27" "28" "29" "30"
#> [31] "31" "32" "33" "34" "35" "36" "37" "38" "39" "40" "41" "42" "43" "44" "45"
#> [46] "46" "47" "48" "49" "50"

Descriptive statistics:

summary(df)            # summary of all variables

#>     State             Undergrads        Population        UGradPerThou  
#>  Length:50          Min.   :  27463   Min.   :  512573   Min.   :37.77  
#>  Class :character   1st Qu.:  90659   1st Qu.: 1771524   1st Qu.:45.73  
#>  Mode  :character   Median : 203641   Median : 4221537   Median :49.43  
#>                     Mean   : 302136   Mean   : 5955551   Mean   :51.45  
#>                     3rd Qu.: 333223   3rd Qu.: 6422700   3rd Qu.:54.46  
#>                     Max.   :2172354   Max.   :36121296   Max.   :77.13

summary(df$Population) # summary of one column

#>     Min.  1st Qu.   Median     Mean  3rd Qu.     Max. 
#>   512573  1771524  4221537  5955551  6422700 36121296

mean(df$Population)    # use a column, NOT mean(df)

#> [1] 5955551

median(df$Population)

#> [1] 4221537

min(df$Population)

#> [1] 512573

max(df$Population)

#> [1] 36121296

TODO (descriptive statistics). Using EG01-19COLLEGE.csv:

1. Calculate the mean of the Undergrads column with mean().
2. Calculate the median of the Undergrads column with median().

mean(df$Undergrads)

#> [1] 302135.9

median(df$Undergrads)

#> [1] 203641

3 Histograms and Boxplots (Base R)

3.1 Histograms (continuous variables)

Histograms show the distribution of a variable — are values concentrated, spread out, or skewed?

hist(df$Population)                # basic

hist(df$Population, breaks = 20)   # adjust number of bins

hist(df$Population, breaks = 100)

hist(df$Population, breaks = 20,
     main = "Population (Histogram)",
     xlab = "Population", ylab = "Frequency", col = "lightgreen")

3.2 Boxplots (quartiles and outliers)

Boxplots show the median, lower and upper quartiles, and outliers.

boxplot(df$Population)                     # basic vertical

boxplot(df$Population, horizontal = TRUE)  # horizontal style

boxplot(df$Population, horizontal = TRUE,
        main = "Population (Boxplot)",
        xlab = "Population", col = "purple")

4 Visualization with ggplot2

Build plots layer by layer: start with data + aes() + geoms. ggplot2 is a grammar of graphics. Remember: (1) use + to add layers, and (2) the order matters.

library(ggplot2)

4.1 Scatterplot (Gapminder data)

Goal: explore the relationship between GDP per capita and life expectancy.

library(gapminder)
gm <- gapminder

names(gm)

#> [1] "country"   "continent" "year"      "lifeExp"   "pop"       "gdpPercap"

ggplot(gm, aes(x = gdpPercap, y = lifeExp))   # base layer only

ggplot(data = gm, aes(x = gdpPercap, y = lifeExp)) +
  geom_point()                                 # add scatter points

ggplot(data = gm, aes(x = gdpPercap, y = lifeExp)) +
  geom_point() +
  labs(title = "Life Expectancy vs GDP per Capita",
       x = "GDP per Capita", y = "Life Expectancy")

ggplot(data = gm, aes(x = gdpPercap, y = lifeExp)) +
  geom_point() +
  labs(title = "Life Expectancy vs GDP per Capita",
       x = "GDP per Capita", y = "Life Expectancy") +
  geom_smooth(color = "pink")                  # smooth trend line

# Note: grey shading = confidence interval

4.2 Histogram (continuous variable)

Show the distribution of Population from EG01-19COLLEGE.csv.

ggplot(data = df, aes(x = Population)) +
  geom_histogram()                             # basic histogram

ggplot(data = df, aes(x = Population)) +
  geom_histogram(bins = 40)                    # adjust number of bins

ggplot(df, aes(x = Population)) +
  geom_histogram(bins = 40, fill = "steelblue") +
  labs(title = "Population (Histogram)", x = "Population", y = "Count")

4.3 Bar Chart (categorical variable)

Goal: draw a bar chart using the EG01-07ONLINE.rda dataset.

load("EG01-07ONLINE.rda")
df2 <- `EG01-07ONLINE`   # rename the data frame

ggplot(df2, aes(x = Source, y = Count)) +
  geom_bar(stat = "identity")                  # basic bar chart

ggplot(data = df2, aes(x = Source, y = Count)) +
  geom_bar(stat = "identity", fill = "forestgreen") +
  labs(title = "Source (Bar Chart)", x = "Source", y = "Counts")

# Change the order of the bars
order <- c("Library", "Google", "Wikipedia", "Other")

ggplot(data = df2, aes(x = Source, y = Count)) +
  geom_bar(stat = "identity", fill = "forestgreen") +
  labs(title = "Source (Bar Chart)", x = "Source", y = "Counts") +
  scale_x_discrete(limits = order)

TODO (visualization). Using the gapminder data, draw a bar chart of continent (x) vs. lifeExp (y). Pick your favorite fill color, add a title and axis labels with labs(), and reorder the continents as: Americas, Asia, Africa, Oceania, Europe.

order2 <- c("Americas", "Asia", "Africa", "Oceania", "Europe")

ggplot(data = gm, aes(x = continent, y = lifeExp)) +
  geom_bar(stat = "identity", fill = "brown") +
  labs(title = "Continent vs. Life Expectancy",
       x = "Continent", y = "Life Expectancy") +
  scale_x_discrete(limits = order2)

5 Variable Transformation

gpa <- read.csv("EX16-19GPA.csv")

knitr::kable(head(gpa), caption = "First rows of the GPA data")

First rows of the GPA data

obs	GPA	HSM	HSS	HSE	SATM	SATCR	SATW	sex
1	3.84	10	10	10	630	570	590	2
2	3.97	10	10	10	750	700	630	1
3	3.49	8	10	9	570	510	490	2
4	1.95	6	4	8	640	600	610	1
5	2.59	8	10	9	510	490	490	2
6	3.00	7	10	10	660	680	630	1

Codebook.

• GPA = grade-point average
• HSM, HSS, HSE = high-school math / science / English grades
• SATM, SATCR, SATW = SAT Math / Critical Reading / Writing
• sex = 1 male, 2 female

A log transformation is often used to reduce skewness. For example, create a log version of the SAT Math scores:

gpa$logSATM <- log(gpa$SATM)

hist(gpa$SATM, breaks = 20)

hist(gpa$logSATM)

The log makes large numbers more comparable and can linearize certain relationships.

TODO (log transformation). Make a log version of the SAT Writing score and compare histograms of the raw and log-transformed values.

gpa$logSATW <- log(gpa$SATW)

hist(gpa$SATW, breaks = 20)

hist(gpa$logSATW)

6 Correlation

Correlation measures how two variables move together.

# Scatterplot: SAT Writing vs. SAT Critical Reading
ggplot(data = gpa, aes(x = SATW, y = SATCR)) +
  geom_point()

# Correlation coefficient
cor(gpa$SATW, gpa$SATCR, use = "complete.obs")

#> [1] 0.7343451

A common threshold for high correlation is 0.7 or above.

TODO (correlation). Draw a scatterplot of GPA vs. HSM (high-school math grades) and calculate the correlation coefficient.

ggplot(data = gpa, aes(x = GPA, y = HSM)) +
  geom_point()

cor(gpa$GPA, gpa$HSM, use = "complete.obs")

#> [1] 0.4197389

7 Linear Regression

Regression is about predicting a relationship — drawing the best-fit line through the data. We estimate it with lm(y ~ x, data = ).

# Predict SAT Critical Reading from SAT Writing
m1 <- lm(SATCR ~ SATW, data = gpa)
# Note: use a new name for each model, or it will overwrite the old one.

summary(m1)

#> 
#> Call:
#> lm(formula = SATCR ~ SATW, data = gpa)
#> 
#> Residuals:
#>      Min       1Q   Median       3Q      Max 
#> -225.643  -35.408    0.169   39.678  152.563 
#> 
#> Coefficients:
#>              Estimate Std. Error t value Pr(>|t|)    
#> (Intercept) 121.79546   34.68694   3.511 0.000591 ***
#> SATW          0.80342    0.06104  13.161  < 2e-16 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Residual standard error: 59.68 on 148 degrees of freedom
#> Multiple R-squared:  0.5393, Adjusted R-squared:  0.5361 
#> F-statistic: 173.2 on 1 and 148 DF,  p-value: < 2.2e-16

# Scatterplot + regression line
ggplot(gpa, aes(x = SATW, y = SATCR)) +
  geom_point() +
  geom_smooth(method = lm, color = "red")

# Gray shaded area: confidence interval (the margin of uncertainty)

TODO (regression). Predict SATCR from SATM, interpret the output, then plot the scatterplot with the regression line added.

m2 <- lm(SATCR ~ SATM, data = gpa)
summary(m2)

#> 
#> Call:
#> lm(formula = SATCR ~ SATM, data = gpa)
#> 
#> Residuals:
#>      Min       1Q   Median       3Q      Max 
#> -275.264  -38.826   -6.939   46.023  202.889 
#> 
#> Coefficients:
#>              Estimate Std. Error t value Pr(>|t|)    
#> (Intercept) 150.93431   49.27543   3.063   0.0026 ** 
#> SATM          0.67810    0.07846   8.643 8.24e-15 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Residual standard error: 71.67 on 148 degrees of freedom
#> Multiple R-squared:  0.3354, Adjusted R-squared:  0.3309 
#> F-statistic:  74.7 on 1 and 148 DF,  p-value: 8.241e-15

ggplot(gpa, aes(x = SATM, y = SATCR)) +
  geom_point() +
  geom_smooth(method = lm, color = "red")