13-02: Simple Linear Regression¶
Correlation (13-01) measures how strong the linear relationship is. Regression writes down the line itself, so you can describe the relationship precisely and predict y from x.
The Regression Line¶
ŷ = b₀ + b₁ x
ŷ the PREDICTED value of y (read "y-hat")
b₀ the y-intercept — predicted y when x = 0
b₁ the slope — the change in ŷ for a ONE-UNIT increase in x
x is the independent / explanatory / predictor variable; y is the dependent / response variable. Swapping them gives a different line, unlike correlation.
The Least-Squares Criterion¶
A residual is the vertical distance from a point to the line:
The least-squares line is the unique line that minimizes Σe², the sum of squared residuals. Squaring is what makes the solution unique and connects it to the mean (03-01, property 3).
Σ (x − x̄)(y − ȳ) n(Σxy) − (Σx)(Σy) s_y
b₁ = ─────────────────── = ─────────────────────── = r · ─────
Σ (x − x̄)² n(Σx²) − (Σx)² s_x
b₀ = ȳ − b₁ x̄
Two facts that follow immediately:
- The line always passes through
(x̄, ȳ), the point of means. - The residuals always sum to zero:
Σe = 0.
The Coefficient of Determination¶
SST = Σ (y − ȳ)² total variation in y
SSR = Σ (ŷ − ȳ)² variation EXPLAINED by the regression
SSE = Σ (y − ŷ)² variation left UNEXPLAINED (residual)
SST = SSR + SSE
SSR SSE
r² = ─────── = 1 − ─────── 0 ≤ r² ≤ 1
SST SST
For simple linear regression, r² = (correlation r)² exactly.
The standard error of the estimate¶
s_e is the typical size of a prediction error, in the units of y — the most interpretable measure of how good the model is. Roughly 95% of observations fall within ±2 s_e of the line.
Worked Example¶
Study hours vs. exam score, the data from 13-01:
Slope and intercept
8(4563) − (56)(602) 36504 − 33712 2792
b₁ = ───────────────────── = ─────────────── = ────── = 3.7935
8(484) − 56² 3872 − 3136 736
b₀ = 75.25 − 3.7935 × 7.0 = 75.25 − 26.554 = 48.696
The fitted line
Interpretation
- Slope: each additional hour of study is associated with an increase of about 3.79 points in predicted exam score.
- Intercept: a student who studies 0 hours is predicted to score about 48.7. Here
x = 0is just outside the observed range (2–12 hours), so this is a mild extrapolation — treat it as a mathematical anchor, not a claim.
Prediction
x = 7 hours → ŷ = 48.696 + 3.7935(7) = 48.696 + 26.555 = 75.25 points
x = 10 hours → ŷ = 48.696 + 3.7935(10) = 48.696 + 37.935 = 86.63 points
x = 40 hours → EXTRAPOLATION — do not do this. The observed range is 2 to 12.
Residual for student 3 (x = 5, y = 68):
ŷ = 48.696 + 3.7935(5) = 48.696 + 18.968 = 67.66
e = 68 − 67.66 = +0.34 the model under-predicted by 0.34 points
Fit statistics
r² = 0.99491² = 0.9898 98.98% of the variation in score is explained
SST = Σ(y − ȳ)² = 1337.50
SSR = 0.9898 × 1337.50 = 1323.90
SSE = SST − SSR = 13.60
______
/ 13.60
s_e = √ ──────── = √2.2667 = 1.506 points
6
Typical prediction error: about 1.5 points. Very tight — this is simulated data; real study-hours data is much noisier.
Prediction vs. Confidence Intervals¶
Two different intervals, often confused:
| Interval | Estimates | Width |
|---|---|---|
Confidence interval for the mean response at x₀ |
The average y for all individuals with that x |
Narrower |
Prediction interval for an individual at x₀ |
The y value of one new individual |
Wider — it carries both the uncertainty in the line and the individual scatter |
_________________________
/ 1 (x₀ − x̄)²
CI for mean : ŷ ± t · s_e · √ ── + ───────────────
n Σ(x − x̄)²
_____________________________
/ 1 (x₀ − x̄)²
PI for one : ŷ ± t · s_e · √ 1 + ── + ───────────────
n Σ(x − x̄)²
Both are narrowest at x₀ = x̄ and widen as you move away — another reason extrapolation is dangerous.
Assumptions of Linear Regression — LINE¶
L LINEARITY the relationship really is linear
I INDEPENDENCE residuals are independent of one another
N NORMALITY residuals are approximately normally distributed
E EQUAL VARIANCE residual spread is constant across x (homoscedasticity)
The diagnostic that checks all four at once is the residual plot: residuals on the vertical axis, x (or ŷ) on the horizontal.
GOOD — random scatter BAD — curvature BAD — fanning out
(heteroscedasticity)
· · · · · · · · ·
──·──·───·──·───·── ──·───·───·────·── ──·──·──·───·────·──
· · · · · · · · ·
· ·
model is fine fit a curve / transform transform y, or use
weighted least squares
Excel¶
' ══ SLOPE, INTERCEPT, FIT ═══════════════════════════════════════════
' Hours in A2:A9, Scores in B2:B9 (y first, then x — note the order!)
=SLOPE(B2:B9, A2:A9) ' b₁ -> 3.79348
=INTERCEPT(B2:B9, A2:A9) ' b₀ -> 48.69565
=RSQ(B2:B9, A2:A9) ' r² -> 0.98984
=CORREL(A2:A9, B2:B9) ' r -> 0.99491
=STEYX(B2:B9, A2:A9) ' s_e -> 1.50554
' ══ PREDICTION ══════════════════════════════════════════════════════
=FORECAST.LINEAR(10, B2:B9, A2:A9) ' ŷ at x = 10 -> 86.63
=TREND(B2:B9, A2:A9, 10) ' same
=INTERCEPT(B2:B9,A2:A9)+SLOPE(B2:B9,A2:A9)*10 ' the same, spelled out
=TREND(B2:B9, A2:A9, A2:A9) ' all fitted values (spills)
' ══ RESIDUALS ═══════════════════════════════════════════════════════
=B2-TREND($B$2:$B$9,$A$2:$A$9,A2) ' C2: residual, fill down
=SUM(C2:C9) ' must be ≈ 0
=SUMSQ(C2:C9) ' SSE -> 13.60
=DEVSQ(B2:B9) ' SST -> 1337.50
=DEVSQ(B2:B9)-SUMSQ(C2:C9) ' SSR -> 1323.90
' ══ LINEST — the whole model in one array formula ═══════════════════
=LINEST(B2:B9, A2:A9, TRUE, TRUE) ' spills a 5×2 block:
' row 1: b₁ b₀
' row 2: SE(b₁) SE(b₀)
' row 3: r² s_e
' row 4: F df
' row 5: SSR SSE
' (pre-365: select a 5×2 range and confirm with Ctrl+Shift+Enter)
=INDEX(LINEST(B2:B9,A2:A9,TRUE,TRUE), 3, 1) ' pull out r²
' ══ ANALYSIS TOOLPAK — the full report ══════════════════════════════
' Data ▸ Data Analysis ▸ Regression
' Input Y Range = B1:B9, Input X Range = A1:A9, tick Labels,
' tick Residuals, Residual Plots, Line Fit Plots, Normal Probability Plots
' → Regression Statistics (Multiple R, R Square, Adjusted R Square,
' Standard Error, Observations)
' → ANOVA table (df, SS, MS, F, Significance F)
' → Coefficients with Standard Error, t Stat, P-value, and 95% CI
' ══ CHART WITH TRENDLINE ════════════════════════════════════════════
' Select A1:B9 ▸ Insert ▸ Scatter ▸ right-click a point ▸ Add Trendline
' ▸ Linear ▸ tick "Display Equation on chart" and "Display R-squared value"
Warning
Excel's regression functions take known_ys first, then known_xs — the opposite of the (x, y) order used everywhere else. SLOPE(A2:A9, B2:B9) silently returns the slope of the inverse regression.
R¶
hours <- c(2, 3, 5, 6, 8, 9, 11, 12)
scores <- c(55, 60, 68, 72, 80, 85, 88, 94)
# ══ FIT THE MODEL ═══════════════════════════════════════════════════
model <- lm(scores ~ hours)
model
# (Intercept) hours
# 48.696 3.793
summary(model)
# Coefficients:
# Estimate Std. Error t value Pr(>|t|)
# (Intercept) 48.69565 1.22090 39.88 1.62e-08 ***
# hours 3.79348 0.15687 24.18 3.30e-07 ***
# Residual standard error: 1.506 on 6 degrees of freedom
# Multiple R-squared: 0.9898, Adjusted R-squared: 0.9881
# F-statistic: 584.7 on 1 and 6 DF, p-value: 3.3e-07
coef(model) # b₀ and b₁
confint(model) # 95% CI for each coefficient
summary(model)$r.squared # 0.98984
summary(model)$sigma # s_e -> 1.5055
anova(model) # the SSR / SSE decomposition
# ══ PREDICTION ══════════════════════════════════════════════════════
predict(model, data.frame(hours = 10)) # 86.63
predict(model, data.frame(hours = 10), interval = "confidence") # mean response
predict(model, data.frame(hours = 10), interval = "prediction") # ONE individual
# fit lwr upr
# 1 86.6304 82.5564 90.7044 (prediction — wider)
# (confidence interval for the mean is 84.8915 to 88.3693 — narrower)
fitted(model) # all ŷ
residuals(model) # all e = y − ŷ
sum(residuals(model)) # ≈ 0
sum(residuals(model)^2) # SSE -> 13.60
# ══ PLOTTING ════════════════════════════════════════════════════════
plot(hours, scores, pch = 19, col = "#5B2A86", cex = 1.4,
xlab = "Hours studied", ylab = "Exam score")
abline(model, col = "#0FA3A3", lwd = 2)
text(4, 90, sprintf("y = %.2f + %.2f x\nR² = %.4f",
coef(model)[1], coef(model)[2], summary(model)$r.squared))
# With confidence and prediction bands
newx <- data.frame(hours = seq(2, 12, 0.1))
ci <- predict(model, newx, interval = "confidence")
pi <- predict(model, newx, interval = "prediction")
lines(newx$hours, ci[, "lwr"], col = "#0FA3A3", lty = 2)
lines(newx$hours, ci[, "upr"], col = "#0FA3A3", lty = 2)
lines(newx$hours, pi[, "lwr"], col = "#8A5FBF", lty = 3)
lines(newx$hours, pi[, "upr"], col = "#8A5FBF", lty = 3)
# ══ CHECKING THE LINE ASSUMPTIONS ═══════════════════════════════════
par(mfrow = c(2, 2))
plot(model) # 1 Residuals vs Fitted (linearity, equal variance)
# 2 Normal Q-Q (normality of residuals)
# 3 Scale-Location (equal variance)
# 4 Residuals vs Leverage(influential points, Cook's distance)
par(mfrow = c(1, 1))
shapiro.test(residuals(model)) # normality
library(car); ncvTest(model) # non-constant variance
durbinWatsonTest(model) # independence (serial correlation)
Python¶
import numpy as np
import pandas as pd
import statsmodels.api as sm
from scipy import stats
import matplotlib.pyplot as plt
hours = np.array([2, 3, 5, 6, 8, 9, 11, 12])
scores = np.array([55, 60, 68, 72, 80, 85, 88, 94])
# ══ QUICK FIT (scipy) ═══════════════════════════════════════════════
res = stats.linregress(hours, scores)
res.slope, res.intercept, res.rvalue, res.pvalue, res.stderr
# 3.79348, 48.69565, 0.99491, 3.30e-07, 0.15687
res.rvalue ** 2 # r² -> 0.98984
# ══ FULL MODEL (statsmodels — the R-style report) ═══════════════════
X = sm.add_constant(hours) # adds the intercept column
model = sm.OLS(scores, X).fit()
print(model.summary())
# coef std err t P>|t|
# const 48.6957 1.221 39.885 0.000
# x1 3.7935 0.157 24.182 0.000
# R-squared: 0.990 F-statistic: 584.8 Prob (F-statistic): 3.30e-07
model.params # [48.6957, 3.7935]
model.conf_int() # 95% CI for each coefficient
model.rsquared, model.rsquared_adj
np.sqrt(model.mse_resid) # s_e -> 1.5055
model.fittedvalues, model.resid
(model.resid ** 2).sum() # SSE -> 13.60
# Formula interface (needs a DataFrame)
# from statsmodels.formula.api import ols
# ols("scores ~ hours", data=pd.DataFrame({...})).fit().summary()
# ══ PREDICTION ══════════════════════════════════════════════════════
model.predict([1, 10]) # ŷ at x = 10 -> 86.63
pred = model.get_prediction([1, 10])
pred.summary_frame(alpha=0.05)
# mean mean_se mean_ci_lower mean_ci_upper obs_ci_lower obs_ci_upper
# 86.6304 0.7107 84.8915 88.3693 82.5564 90.7044
# └── confidence interval (mean) ──┘ └── prediction interval ──┘
# ══ PLOTTING ════════════════════════════════════════════════════════
fig, ax = plt.subplots()
ax.scatter(hours, scores, s=70, color="#5B2A86")
xs = np.linspace(2, 12, 100)
ax.plot(xs, res.intercept + res.slope * xs, color="#0FA3A3", lw=2)
ax.set(xlabel="Hours studied", ylabel="Exam score")
ax.text(3, 90, f"y = {res.intercept:.2f} + {res.slope:.2f}x\nR² = {res.rvalue**2:.4f}")
# ══ RESIDUAL DIAGNOSTICS ════════════════════════════════════════════
fig, axes = plt.subplots(1, 3, figsize=(13, 3.5))
axes[0].scatter(model.fittedvalues, model.resid, color="#5B2A86")
axes[0].axhline(0, ls="--", color="grey")
axes[0].set(xlabel="Fitted", ylabel="Residual", title="Residuals vs Fitted")
sm.qqplot(model.resid, line="s", ax=axes[1])
axes[1].set_title("Normal Q-Q")
axes[2].hist(model.resid, bins=6, color="#0FA3A3", edgecolor="white")
axes[2].set_title("Residual histogram")
plt.tight_layout(); plt.show()
stats.shapiro(model.resid)
sm.stats.diagnostic.het_breuschpagan(model.resid, model.model.exog) # equal variance
sm.stats.durbin_watson(model.resid) # independence
Quick Reference¶
| Task | Excel | R | Python |
|---|---|---|---|
| Slope | SLOPE(y, x) |
coef(lm(y~x))[2] |
linregress(x,y).slope |
| Intercept | INTERCEPT(y, x) |
coef(lm(y~x))[1] |
linregress(x,y).intercept |
r² |
RSQ(y, x) |
summary(m)$r.squared |
res.rvalue**2 |
Standard error s_e |
STEYX(y, x) |
summary(m)$sigma |
np.sqrt(m.mse_resid) |
| Predict | FORECAST.LINEAR(x₀,y,x) |
predict(m, newdata) |
m.predict([1, x0]) |
| Fitted values | TREND(y, x, x) |
fitted(m) |
m.fittedvalues |
| Residuals | y - TREND(...) |
residuals(m) |
m.resid |
| Full model | LINEST / ToolPak ▸ Regression |
summary(lm(y~x)) |
sm.OLS(y,X).fit().summary() |
| Coefficient CIs | ToolPak Regression output | confint(m) |
m.conf_int() |
| Prediction interval | manual formula | predict(..., interval="prediction") |
get_prediction().summary_frame() |
| Diagnostics | ToolPak residual plots | plot(m) |
manual + sm.qqplot |
Common Mistakes¶
- Swapping the argument order in Excel:
SLOPE(known_y, known_x), y first. - Extrapolating beyond the observed range of
x. - Interpreting the intercept when
x = 0is far outside the data or physically impossible. - Interpreting the slope causally. Regression describes association (13-01).
- Reporting
r²without ever plotting the residuals — a curved relationship can still show a highr². - Confusing a confidence interval for the mean response with a prediction interval for an individual.
- Regressing
yonxand then reading the line backwards to predictxfromy. That is a different regression. - Fitting a line to data with an influential outlier without checking leverage or Cook's distance.
Exercises: 13-02: Exercises — Simple Linear Regression
⬅️ Previous: 13-01: Correlation ➡️ Next: 13-03: Regression Inference and Multiple Regression