13-02: Exercises — Simple Linear Regression¶
Notes reference: 13-02: Simple Linear Regression
Q1: Fit the line by hand¶
Advertising spend (x, $000s) and sales (y, $000s), the data from 13-01:
Find b₁, b₀, and the fitted line.
Solution
n(Σxy) − (Σx)(Σy) 6(1935) − 36(290) 11610 − 10440 1170
b₁ = ─────────────────── = ────────────────── = ───────────── = ──────
n(Σx²) − (Σx)² 6(258) − 36² 1548 − 1296 252
= 4.642857
x̄ = 36/6 = 6.0 ȳ = 290/6 = 48.333333
b₀ = ȳ − b₁x̄ = 48.333333 − 4.642857(6.0) = 48.333333 − 27.857143 = 20.476190
FITTED LINE: ŷ = 20.4762 + 4.6429 x
INTERPRETATION
SLOPE Each additional $1,000 of advertising is associated with an
increase of about $4,643 in sales.
INTERCEPT With zero advertising, predicted sales are about $20,476.
Here x = 0 is just below the observed range (2 to 10), so
this is a mild extrapolation — plausible but not established.
x <- c(2,4,5,7,8,10); y <- c(30,38,45,52,58,67)
coef(lm(y ~ x))
# (Intercept) x
# 20.476190 4.642857
Q2: Predict and compute residuals¶
Using ŷ = 20.4762 + 4.6429x:
- Predict sales for
x = 6. - Predict sales for
x = 9. - Find the residual for the observation
(5, 45). - Should you predict for
x = 25?
Solution
1. ŷ = 20.4762 + 4.6429(6) = 20.4762 + 27.8571 = 48.3333 ($48,333)
Note this equals ȳ exactly — the line always passes through (x̄, ȳ).
2. ŷ = 20.4762 + 4.6429(9) = 20.4762 + 41.7857 = 62.2619 ($62,262)
3. ŷ = 20.4762 + 4.6429(5) = 20.4762 + 23.2143 = 43.6905
e = y − ŷ = 45 − 43.6905 = +1.3095
The model UNDER-predicted this month by about $1,310.
4. NO. The observed x range is 2 to 10. Predicting at x = 25 is
EXTRAPOLATION far outside the data. The linear relationship has
never been observed there, and sales almost certainly saturate at
high spend rather than continuing to rise by $4,643 per $1,000.
=FORECAST.LINEAR(9, B2:B7, A2:A7) ' 62.2619
=TREND(B2:B7, A2:A7, 9) ' same
=B4-TREND($B$2:$B$7,$A$2:$A$7,A4) ' residual for row 4 -> 1.3095
m <- lm(y ~ x)
predict(m, data.frame(x = c(6, 9))) # 48.3333 62.2619
residuals(m) # all six residuals
sum(residuals(m)) # ≈ 0 — always
Q3: Full fit statistics¶
For the Q1 data, compute SST, SSR, SSE, r², and s_e.
Solution
FITTED VALUES AND RESIDUALS
x y ŷ = 20.4762 + 4.6429x e = y − ŷ e²
2 30 29.7619 +0.2381 0.0567
4 38 39.0476 −1.0476 1.0975
5 45 43.6905 +1.3095 1.7149
7 52 52.9762 −0.9762 0.9529
8 58 57.6190 +0.3810 0.1451
10 67 66.9048 +0.0952 0.0091
──────── ────────
0.0000 3.9762
SSE = Σe² = 3.9762 df = n − 2 = 4
SST = Σ(y − ȳ)²
= (30−48.3333)² + (38−48.3333)² + (45−48.3333)²
+ (52−48.3333)² + (58−48.3333)² + (67−48.3333)²
= 336.1111 + 106.7778 + 11.1111 + 13.4444 + 93.4444 + 348.4444
= 909.3333
SSR = SST − SSE = 909.3333 − 3.9762 = 905.3571
SSR 905.3571
r² = ───── = ────────── = 0.99563
SST 909.3333
✓ This matches r² = (0.99781)² from 13-01 exactly, as it must.
_______ ________
/ SSE / 3.9762
s_e = √ ─────── = √ ─────── = √0.99405 = 0.9970
n − 2 4
INTERPRETATION of s_e: a typical prediction misses actual sales by about
$997. Roughly 95% of observations lie within ±2(0.9970) = ±$1,994 of
the line.
=DEVSQ(B2:B7) ' SST -> 909.3333
=SUMSQ(C2:C7) ' SSE (residuals in C) -> 3.9762
=DEVSQ(B2:B7)-SUMSQ(C2:C7) ' SSR -> 905.3571
=RSQ(B2:B7, A2:A7) ' r² -> 0.995628
=STEYX(B2:B7, A2:A7) ' s_e -> 0.997019
m <- lm(y ~ x)
summary(m)$r.squared # 0.995628
summary(m)$sigma # 0.997019 = s_e
anova(m) # SSR, SSE, and the F-test
sum(residuals(m)^2) # SSE
Q4: A regression with real scatter¶
Hours of exercise per week (x) and resting heart rate (y) for 10 adults:
x |
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|---|
y |
78 | 76 | 74 | 73 | 70 | 69 | 68 | 65 | 64 | 61 |
Fit the line, interpret it, and give r².
Solution
Σx = 45, Σy = 698, Σxy = 2992, Σx² = 285, n = 10
x̄ = 4.5, ȳ = 69.8
10(2992) − 45(698) 29920 − 31410 −1490
b₁ = ──────────────────── = ─────────────── = ────── = −1.806061
10(285) − 45² 2850 − 2025 825
b₀ = 69.8 − (−1.806061)(4.5) = 69.8 + 8.127273 = 77.927273
FITTED LINE: ŷ = 77.9273 − 1.8061 x
INTERPRETATION
SLOPE each additional hour of weekly exercise is associated with a
DECREASE of about 1.81 beats per minute in resting heart rate
INTERCEPT someone who does no exercise is predicted to have a resting
heart rate of about 77.9 bpm. Here x = 0 IS in the observed
range, so the intercept is directly meaningful.
r = −0.99552 (strong negative)
r² = 0.99107 → 99.1% of the variation in heart rate is explained
by hours of exercise
s_e = 0.5587 bpm → predictions are typically within about 0.6 bpm
x <- 0:9; y <- c(78,76,74,73,70,69,68,65,64,61)
m <- lm(y ~ x)
summary(m)
plot(x, y, pch = 19, col = "#5B2A86", cex = 1.4,
xlab = "Hours of exercise per week", ylab = "Resting heart rate (bpm)")
abline(m, col = "#0FA3A3", lwd = 2)
CAUSATION WARNING
This is observational data. Exercise MAY lower heart rate, but fitter
people also exercise more (reverse causation), and age, weight, and
smoking plausibly drive both. Only a randomized trial establishes cause.
Q5: Interpret a LINEST block¶
Identify every entry.
Solution
Row 1: b₁ = 4.6429 b₀ = 20.4762
Row 2: SE(b₁) = 0.1538 SE(b₀) = 1.0088
Row 3: r² = 0.9956 s_e = 0.9970
Row 4: F = 910.79 df_residual = 4
Row 5: SSR = 905.3571 SSE = 3.9762
WHAT YOU CAN DERIVE
t for the slope = b₁ / SE(b₁) = 4.6429/0.1538 = 30.18
p for the slope = T.DIST.2T(30.18, 4) = 7.2e-06
Check: F = t² → 30.18² = 910.8 ✓ (as it must be for one predictor)
95% CI for β₁ = 4.6429 ± T.INV.2T(0.05,4) × 0.1538
= 4.6429 ± 2.7764(0.1538) = (4.2159, 5.0699)
Check: SSR + SSE = 905.3571 + 3.9762 = 909.3333 = SST ✓
NOTE ON ORDER
LINEST returns coefficients RIGHT TO LEFT — the intercept is the
RIGHTMOST cell, not the leftmost. With several predictors the slopes
also come out reversed. Prefer the Analysis ToolPak's Regression
output, which lists them in the natural order.
=LINEST(B2:B7, A2:A7, TRUE, TRUE) ' the whole 5×2 block
=INDEX(LINEST(B2:B7,A2:A7,TRUE,TRUE), 2, 1) ' SE(b₁)
=INDEX(LINEST(B2:B7,A2:A7,TRUE,TRUE), 3, 1) ' r²
Q6: Residual plots¶
Sketch what each residual plot means and say what to do.
- Random scatter around 0
- A clear U shape
- A funnel widening to the right
- One point far from the rest
- A cyclical wave pattern
Solution
1. RANDOM SCATTER — the model is fine. Linearity, equal variance, and
independence all look satisfied. Proceed.
2. U SHAPE (curvature) — LINEARITY IS VIOLATED. A straight line is the
wrong model.
FIX: add a quadratic term (y ~ x + I(x^2)), transform x or y, or fit
a nonlinear model.
3. FUNNEL — HETEROSCEDASTICITY (unequal variance). The spread of errors
grows with the fitted value, so predictions are much less reliable at
the high end and the standard errors are wrong.
FIX: log-transform y, use weighted least squares, or use
heteroscedasticity-robust standard errors.
4. ONE FAR POINT — an OUTLIER or an INFLUENTIAL point.
FIX: investigate it. Check leverage and Cook's distance. Report the
fit with and without it.
5. CYCLICAL WAVE — the residuals are AUTOCORRELATED (a violation of
independence), typical of time-series data.
FIX: add a time or seasonal term, or model the errors explicitly
(ARIMA). Check with the Durbin-Watson statistic.
par(mfrow = c(2, 2)); plot(m); par(mfrow = c(1, 1))
# residuals vs fitted, by hand
plot(fitted(m), residuals(m), pch = 19, col = "#5B2A86",
xlab = "Fitted", ylab = "Residual")
abline(h = 0, lty = 2, col = "grey40")
library(car)
ncvTest(m) # non-constant variance test
durbinWatsonTest(m) # independence
which(cooks.distance(m) > 4/length(y)) # influential points
' Data ▸ Data Analysis ▸ Regression
' tick Residuals, Standardized Residuals, Residual Plots,
' Line Fit Plots, Normal Probability Plots
' The Residual Plot is exactly the diagnostic described above.
Q7: Confidence vs. prediction interval¶
For the Q1 regression at x₀ = 7, build both a 95% confidence interval for the mean response and a 95% prediction interval for one new observation.
Solution
ŷ at x₀ = 7: 20.4762 + 4.6429(7) = 52.9762
s_e = 0.9970, n = 6, x̄ = 6.0, Σ(x − x̄)² = 252/6 = 42
t(0.025, 4) = 2.7764
(x₀ − x̄)² = (7 − 6)² = 1
CONFIDENCE INTERVAL for the MEAN response at x = 7
_________________
/ 1 (x₀ − x̄)²
ŷ ± t · s_e · √ ── + ────────────
n Σ(x − x̄)²
= 52.9762 ± 2.7764(0.9970)·√(1/6 + 1/42)
= 52.9762 ± 2.7764(0.9970)·√(0.166667 + 0.023810)
= 52.9762 ± 2.7764(0.9970)(0.436437)
= 52.9762 ± 1.2081
= (51.768, 54.184)
PREDICTION INTERVAL for ONE new observation at x = 7
______________________
/ 1 (x₀ − x̄)²
ŷ ± t · s_e · √ 1 + ── + ────────────
n Σ(x − x̄)²
= 52.9762 ± 2.7764(0.9970)·√(1 + 0.166667 + 0.023810)
= 52.9762 ± 2.7764(0.9970)(1.090627)
= 52.9762 ± 3.0190
= (49.957, 55.995)
COMPARE
CI width = 2.42 ← the plausible range for the AVERAGE sales
of all months with $7,000 of advertising
PI width = 6.04 ← the plausible range for ONE such month
The prediction interval is 2.5× wider, because it carries BOTH the
uncertainty in the line AND the individual month-to-month scatter.
NEVER quote a confidence interval when the question is about a single
future observation.
m <- lm(y ~ x)
predict(m, data.frame(x = 7), interval = "confidence")
# fit lwr upr
# 52.9762 51.7681 54.1843
predict(m, data.frame(x = 7), interval = "prediction")
# fit lwr upr
# 52.9762 49.9572 55.9952
Q8: Regression with a transformation¶
Bacteria counts double roughly every hour:
Hour x |
0 | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|---|
Count y |
100 | 210 | 395 | 810 | 1580 | 3200 |
Fit a straight line, then fit a line to log(y). Compare.
Solution
DIRECT LINEAR FIT (y on x)
b₁ = 572.14, b₀ = −381.19, r² = 0.8186, s_e = 563.3
Problems:
• the intercept is NEGATIVE — impossible for a bacteria count
• the residual plot shows a clear U shape (curvature)
• predictions at low x are badly wrong
LOG-TRANSFORMED FIT (log₁₀ y on x)
log₁₀ y: 2.0000, 2.3222, 2.5966, 2.9085, 3.1987, 3.5051
b₁ = 0.29906, b₀ = 2.00755, r² = 0.99971, s_e = 0.01061
FITTED: log₁₀ ŷ = 2.00755 + 0.29906 x
BACK-TRANSFORMED: ŷ = 10^2.00755 × (10^0.29906)^x
= 101.75 × (1.9917)^x
INTERPRETATION
Initial count ≈ 102 bacteria ✓ matches the observed 100
Growth factor ≈ 1.99 per hour → the population DOUBLES every hour ✓
r² rose from 0.819 to 0.9997, and s_e (in log units) is tiny.
THE LESSON
A straight line through curved data can still show a respectable r²
(0.819 here) while being structurally wrong. The RESIDUAL PLOT is
what exposes it. Transform first, then fit.
x <- 0:5; y <- c(100, 210, 395, 810, 1580, 3200)
m1 <- lm(y ~ x); summary(m1)$r.squared # 0.8186
m2 <- lm(log10(y) ~ x); summary(m2)$r.squared # 0.9997
par(mfrow = c(2, 2))
plot(x, y, pch = 19, col = "#5B2A86", main = "Raw"); abline(m1, col = "#0FA3A3")
plot(fitted(m1), residuals(m1), pch = 19, main = "Residuals — raw (U shape!)")
abline(h = 0, lty = 2)
plot(x, log10(y), pch = 19, col = "#5B2A86", main = "Log scale"); abline(m2, col = "#0FA3A3")
plot(fitted(m2), residuals(m2), pch = 19, main = "Residuals — log (random)")
abline(h = 0, lty = 2)
par(mfrow = c(1, 1))
10^coef(m2)[1] # initial count ≈ 101.8
10^coef(m2)[2] # growth factor ≈ 1.99 per hour
=SLOPE(B2:B7, A2:A7) ' raw slope
=RSQ(B2:B7, A2:A7) ' 0.8186
' Add a column C: =LOG10(B2)
=SLOPE(C2:C7, A2:A7) ' 0.29906
=RSQ(C2:C7, A2:A7) ' 0.99971
=10^INTERCEPT(C2:C7,A2:A7) ' 101.75
=10^SLOPE(C2:C7,A2:A7) ' 1.9917 — the hourly growth factor
⬅️ Previous: 13-01: Exercises — Correlation ➡️ Next: 13-03: Exercises — Regression Inference and Multiple Regression