04-01: Measures of Variation¶
Two data sets can share a mean and be nothing alike. Variation — how spread out the values are — is the other half of every summary, and it is the quantity that every inferential test in Chapters 09–13 is built on.
Same centre, completely different risk, reliability, and interpretation.
Range¶
Fast, intuitive, and fragile: it uses only two values, so one outlier defines it entirely.
Variance and Standard Deviation¶
The average squared distance from the mean. Squaring is necessary because the raw deviations always sum to zero.
Population (you have every member)¶
Sample (the usual case)¶
Why n − 1? The sample mean x̄ is itself estimated from the same data, which uses up one degree of freedom: once you know x̄ and any n − 1 deviations, the last one is forced. Dividing by n would systematically underestimate σ²; dividing by n − 1 makes s² an unbiased estimator. The quantity n − 1 reappears as the degrees of freedom of every t-test in Chapter 11.
Computational (shortcut) formula¶
Algebraically identical, easier by hand and numerically what spreadsheets use:
Units¶
Variance is in squared units (dollars², minutes²) — never quote it to a non-statistician. The standard deviation is back in the original units, which is why it is the one you report.
Coefficient of Variation¶
Standard deviation expressed as a percentage of the mean — the only honest way to compare spread across different units or wildly different scales.
Example. Which varies more — salaries (mean $52,000, s = $8,000) or ages (mean 34 years, s = 9 years)?
Ages vary relatively more, even though the salary standard deviation is a thousand times larger in absolute terms.
Note
CV is only meaningful for ratio data with a positive mean. It is undefined for interval data such as temperature in °C (where the zero point is arbitrary).
The Empirical Rule (68–95–99.7)¶
Applies only to bell-shaped (approximately normal) distributions:
μ ± 1σ contains about 68 % of the data
μ ± 2σ contains about 95 % of the data
μ ± 3σ contains about 99.7 % of the data
┌──────────── 99.7% ────────────┐
│ ┌────── 95% ──────┐ │
│ │ ┌── 68% ──┐ │ │
───┴──────┴───┴────┬────┴───┴──────┴───
μ−3σ μ−2σ μ−1σ μ μ+1σ μ+2σ μ+3σ
Example. IQ scores: μ = 100, σ = 15. About 95% of people score between 70 and 130; a score above 145 is beyond 3σ and occurs in roughly 1 person in 750.
Chebyshev's Theorem¶
Works for any distribution, whatever its shape — the price is a much weaker guarantee.
k |
Chebyshev's minimum | Empirical rule (bell-shaped only) |
|---|---|---|
| 2 | at least 75% | about 95% |
| 3 | at least 88.9% | about 99.7% |
| 4 | at least 93.75% | about 99.99% |
Use Chebyshev when the shape is unknown or clearly skewed; use the empirical rule when a histogram looks bell-shaped.
Worked Example¶
Seven delivery times (minutes): 18, 22, 25, 25, 28, 31, 35
n = 7, Σx = 184, x̄ = 184 / 7 = 26.286
x x − x̄ (x − x̄)² x²
18 −8.286 68.65 324
22 −4.286 18.37 484
25 −1.286 1.65 625
25 −1.286 1.65 625
28 1.714 2.94 784
31 4.714 22.22 961
35 8.714 75.94 1225
──── ──────── ──────── ──────
184 0.000 191.43 5028
Definition formula
Computational formula — same answer, no rounding of the mean:
Range = 35 − 18 = 17 minutes
CV = 5.65 / 26.286 × 100 = 21.5 %
Chebyshev at k = 2: at least 75% of delivery times lie within 26.29 ± 2(5.65) = 15.0 to 37.6 minutes. All 7 values do, which is consistent.
Excel¶
' ── Range ───────────────────────────────────────────────────────────
=MAX(A2:A8)-MIN(A2:A8) ' 17
' ── Variance and standard deviation ─────────────────────────────────
=VAR.S(A2:A8) ' SAMPLE variance (÷ n−1) -> 31.905
=STDEV.S(A2:A8) ' SAMPLE SD -> 5.648
=VAR.P(A2:A8) ' POPULATION variance (÷ N) -> 27.347
=STDEV.P(A2:A8) ' POPULATION SD -> 5.229
' Legacy names still accepted: VAR = VAR.S, STDEV = STDEV.S,
' VARP = VAR.P, STDEVP = STDEV.P
' ── Built by hand (to show the formula) ─────────────────────────────
=SUMPRODUCT((A2:A8-AVERAGE(A2:A8))^2)/(COUNT(A2:A8)-1) ' definition form
=(SUMSQ(A2:A8)-SUM(A2:A8)^2/COUNT(A2:A8))/(COUNT(A2:A8)-1) ' computational form
=DEVSQ(A2:A8) ' Σ(x − x̄)² directly -> 191.43
' ── Coefficient of variation ────────────────────────────────────────
=STDEV.S(A2:A8)/AVERAGE(A2:A8) ' as a proportion
=STDEV.S(A2:A8)/AVERAGE(A2:A8)*100 ' as a percent -> 21.5
' ── Empirical rule bounds ───────────────────────────────────────────
=AVERAGE(A2:A8)-1*STDEV.S(A2:A8) & " to " & AVERAGE(A2:A8)+1*STDEV.S(A2:A8)
=AVERAGE(A2:A8)-2*STDEV.S(A2:A8) & " to " & AVERAGE(A2:A8)+2*STDEV.S(A2:A8)
' ── Chebyshev minimum for a given k ─────────────────────────────────
=1-1/D2^2 ' D2 holds k -> 0.75 when k = 2
' ── Grouped data (from a frequency table) ───────────────────────────
=SQRT(SUMPRODUCT(F2:F5,(G2:G5-$J$1)^2)/(SUM(F2:F5)-1)) ' f in F, Xm in G, x̄ in J1
' ── Conditional spread ──────────────────────────────────────────────
=STDEV.S(IF(B2:B100="Biology", A2:A100)) ' array formula (Ctrl+Shift+Enter pre-365)
Warning
STDEV.S vs STDEV.P is the most consequential one-letter choice in the course. Use .S whenever the data is a sample — which is almost always. Excel's Analysis ToolPak "Descriptive Statistics" reports the sample standard deviation.
Analysis ToolPak: Data ▸ Data Analysis ▸ Descriptive Statistics → Summary statistics gives standard deviation, sample variance, range, minimum, maximum, and standard error in one block.
R¶
delivery <- c(18, 22, 25, 25, 28, 31, 35)
# ── Range ──────────────────────────────────────────────────────────
range(delivery) # 18 35
diff(range(delivery)) # 17
# ── Variance and SD (R defaults to the SAMPLE versions) ───────────
var(delivery) # 31.90476
sd(delivery) # 5.648430
# Population versions must be built:
n <- length(delivery)
pop_var <- var(delivery) * (n - 1) / n # 27.34694
pop_sd <- sqrt(pop_var) # 5.229430
# By hand — both formulas
sum((delivery - mean(delivery))^2) / (n - 1) # definition
(sum(delivery^2) - sum(delivery)^2 / n) / (n - 1) # computational
# ── Coefficient of variation ───────────────────────────────────────
cv <- function(x) sd(x) / mean(x) * 100
cv(delivery) # 21.49
# ── Empirical rule bounds ──────────────────────────────────────────
m <- mean(delivery); s <- sd(delivery)
rbind(k1 = m + c(-1, 1) * s,
k2 = m + c(-2, 2) * s,
k3 = m + c(-3, 3) * s)
# proportion actually inside ±1 SD (checks the "bell-shaped" assumption)
mean(abs(delivery - m) <= s)
# ── Chebyshev minimum ──────────────────────────────────────────────
chebyshev <- function(k) 1 - 1 / k^2
chebyshev(2) # 0.75
# ── Other spread measures ──────────────────────────────────────────
IQR(delivery) # interquartile range (see 04-02)
mad(delivery) # median absolute deviation — very robust
# ── Everything at once ─────────────────────────────────────────────
library(psych)
describe(delivery) # includes sd, se, range, skew, kurtosis
# ── By group ───────────────────────────────────────────────────────
tapply(df$score, df$group, sd)
aggregate(score ~ group, data = df, FUN = function(x) c(mean = mean(x), sd = sd(x)))
Python¶
import numpy as np
import pandas as pd
from scipy import stats
delivery = np.array([18, 22, 25, 25, 28, 31, 35])
# ── Range ──────────────────────────────────────────────────────────
delivery.max() - delivery.min() # 17
np.ptp(delivery) # 17 (peak-to-peak)
# ── Variance and SD ────────────────────────────────────────────────
# NumPy defaults to ddof=0 (POPULATION) — you must ask for ddof=1
delivery.var(ddof=1) # 31.9048 sample
delivery.std(ddof=1) # 5.6484 sample
delivery.var() # 27.3469 population (ddof=0)
delivery.std() # 5.2294 population
# pandas defaults to ddof=1 (SAMPLE) — the opposite convention
s = pd.Series(delivery)
s.var(), s.std() # 31.9048, 5.6484
# By hand — both formulas
n = delivery.size
((delivery - delivery.mean()) ** 2).sum() / (n - 1) # definition
((delivery ** 2).sum() - delivery.sum() ** 2 / n) / (n - 1) # computational
# ── Coefficient of variation ───────────────────────────────────────
stats.variation(delivery, ddof=1) * 100 # 21.49
# ── Empirical rule bounds ──────────────────────────────────────────
m, sd = delivery.mean(), delivery.std(ddof=1)
{k: (m - k * sd, m + k * sd) for k in (1, 2, 3)}
np.mean(np.abs(delivery - m) <= sd) # proportion inside ±1 SD
# ── Chebyshev minimum ──────────────────────────────────────────────
chebyshev = lambda k: 1 - 1 / k ** 2
chebyshev(2) # 0.75
# ── Other spread measures ──────────────────────────────────────────
stats.iqr(delivery) # interquartile range
stats.median_abs_deviation(delivery) # MAD
# ── By group ───────────────────────────────────────────────────────
df.groupby("group")["score"].agg(["mean", "std", "count"])
Warning
ddof traps. NumPy's .std() defaults to the population formula; pandas' .std() defaults to the sample formula. Mixing the two in one script is a silent source of wrong answers. State ddof=1 explicitly whenever you mean "sample".
Quick Reference¶
| Measure | Formula | Excel | R | Python |
|---|---|---|---|---|
| Range | max − min |
MAX-MIN |
diff(range(x)) |
np.ptp(x) |
| Sample variance | Σ(x−x̄)²/(n−1) |
VAR.S |
var(x) |
x.var(ddof=1) |
| Sample SD | √s² |
STDEV.S |
sd(x) |
x.std(ddof=1) |
| Population variance | Σ(X−μ)²/N |
VAR.P |
var(x)*(n-1)/n |
x.var(ddof=0) |
| Population SD | √σ² |
STDEV.P |
sqrt(var(x)*(n-1)/n) |
x.std(ddof=0) |
| Σ(x−x̄)² | — | DEVSQ |
sum((x-mean(x))^2) |
((x-x.mean())**2).sum() |
| CV | s/x̄ × 100 |
STDEV.S/AVERAGE*100 |
sd(x)/mean(x)*100 |
stats.variation(x, ddof=1)*100 |
| IQR | Q3 − Q1 |
QUARTILE(.,3)-QUARTILE(.,1) |
IQR(x) |
stats.iqr(x) |
| MAD | median of |x − median| | manual | mad(x) |
stats.median_abs_deviation(x) |
| Chebyshev bound | 1 − 1/k² |
1-1/k^2 |
1-1/k^2 |
1-1/k**2 |
Common Mistakes¶
- Using the population formula (
STDEV.P,ddof=0) on sample data — every downstream test then has the wrong standard error. - Reporting variance instead of standard deviation to a general audience (wrong units).
- Applying the empirical rule to obviously skewed data — use Chebyshev instead.
- Comparing standard deviations across different units or scales; use the CV.
- Forgetting that the range grows with sample size, so it cannot compare data sets of different
n.
Exercises: 04-01: Exercises — Measures of Variation
⬅️ Previous: 03-02: Weighted and Grouped Means ➡️ Next: 04-02: Measures of Position and Outliers