08-01: Sampling Methods and Bias

Every inference in the rest of this course assumes the sample fairly represents the population. No amount of statistical machinery repairs a badly drawn sample — this note is about getting that part right.

---

Why Sample at All?

Reason	Example
Cost	Polling 1,200 voters instead of 200 million
Time	Results this week, not next year
Destructive testing	You cannot crash-test every car you sell
Inaccessible population	All future output of a production line
Accuracy	A careful sample often beats a rushed census with poor coverage

---

Probability Sampling Methods

In a probability sample, every member of the population has a known, non-zero chance of selection. Only probability samples support the inference methods of Chapters 09–13.

1. Simple random sampling (SRS)

Every possible sample of size n is equally likely.

Method:  number every member 1…N, then draw n numbers at random
         (random number generator, or a random number table)

Pros: unbiased, mathematically simplest — it is the assumption behind every formula in this course. Cons: needs a complete list of the population (a sampling frame); can be expensive to reach a scattered sample.

2. Systematic sampling

Pick every k-th member after a random start.

k = N / n           (the sampling interval)
Random start r between 1 and k, then select r, r+k, r+2k, …

Pros: simple to execute in the field, spreads across the frame. Cons: dangerous if the list has a periodic pattern matching k — e.g. every 10th house on a street where every 10th house is a corner lot.

3. Stratified sampling

Divide the population into homogeneous, non-overlapping strata (age band, region, department), then draw a random sample from each.

Proportional allocation:   nᵢ  =  n × (Nᵢ / N)

Pros: guarantees representation of every subgroup; usually gives smaller standard errors than SRS. Cons: requires knowing the strata in advance.

4. Cluster sampling

Divide the population into naturally occurring clusters (schools, city blocks, flights), randomly select whole clusters, and survey everyone (or a sample) in them.

Pros: far cheaper when the population is geographically spread; needs a list of clusters, not of individuals. Cons: less precise than SRS at the same n, because members of a cluster tend to be similar.

Tip

Stratified vs. cluster — the single most-tested distinction in the chapter:

	Stratified	Cluster
Groups are	Homogeneous within, different between	Heterogeneous within, similar between
You sample	Some from every group	All from some groups
Goal	Precision	Cost
Example	Sample 50 from each grade level	Randomly pick 8 classrooms, survey all in them

5. Multistage sampling

Combine methods: randomly pick states → within each, pick districts → within each, pick households. Standard practice for national surveys.

---

Non-Probability Sampling (Use With Care)

Method	Description	Problem
Convenience	Whoever is easy to reach	Almost certainly unrepresentative
Voluntary response	People opt in (online polls, call-ins)	Strong opinions are over-represented
Judgment / purposive	Researcher picks "typical" cases	Researcher bias
Quota	Fill preset category counts non-randomly	Looks stratified, is not random
Snowball	Participants recruit others	Fine for hidden populations, not for estimation

These can be useful for exploration, but the confidence intervals and p-values of later chapters are not valid for them.

---

Sampling Error and Bias — Not the Same Thing

Sampling error   The natural difference between a sample statistic and the
                 population parameter, caused purely by chance.
                 → Unavoidable. Measurable. SHRINKS as n grows.

Bias             A systematic tendency to miss in one direction.
                 → Avoidable by design. NOT fixed by a larger sample.

Warning

A larger sample makes a biased estimate more precisely wrong. The 1936 Literary Digest poll surveyed 2.4 million people and still called the US election wrong, because its frame (car and telephone owners) skewed wealthy.

Common sources of bias

Bias	Cause	Example
Selection / coverage	Frame excludes part of the population	Landline-only phone survey
Nonresponse	Those who refuse differ from those who answer	8% response rate on a satisfaction survey
Response	Untruthful or influenced answers	Under-reporting alcohol use
Question wording	Leading or loaded phrasing	"Do you support the wasteful new tax?"
Survivorship	Only the surviving cases are observed	Rating funds using only those still trading
Undercoverage	A subgroup is systematically under-sampled	Online-only survey missing older respondents

---

Observational Study vs. Designed Experiment

Observational study   Observe and measure; do NOT assign treatments.
                      → Can establish ASSOCIATION, never causation.

Designed experiment   Deliberately assign treatments to subjects.
                      → With randomization, CAN establish causation.

Principles of experimental design

1. Control — hold other variables constant; include a control or placebo group.
2. Randomization — assign subjects to treatments at random, which balances unknown confounders on average.
3. Replication — enough subjects per group for the result to be more than noise.
4. Blinding — single-blind (subjects unaware) or double-blind (subjects and assessors unaware).

Confounding variable: a variable related to both the treatment and the response, so their effects cannot be separated. Ice-cream sales and drowning deaths correlate; temperature is the confounder. Randomization is the main defence.

Blocking: the experimental analogue of stratification — group similar subjects into blocks, then randomize within each block.

---

How Big Should the Sample Be?

Derived properly in 09-01, but the shape is worth seeing now:

                                       2                                  2
For a mean:        n  =  ( z·σ / E )        For a proportion:  n = p̂q̂(z/E)

Two consequences that surprise people:

• Precision improves with √n, not n. Halving the margin of error needs four times the sample.
• The population size barely matters. A sample of 1,000 is about as accurate for a country of 300 million as for a town of 30,000 — the finite population correction only bites when n > 0.05N.

---

Excel

' ── Simple random sample ────────────────────────────────────────────
' Add a helper column of random numbers, sort by it, take the top n
=RAND()                                  ' put in C2, fill down, then sort by C
=RANDBETWEEN(1, 500)                      ' a random ID (may repeat)
=SORTBY(A2:B501, RANDARRAY(500))          ' 365: shuffle the whole block
=INDEX(SORTBY(A2:A501, RANDARRAY(500)), SEQUENCE(30))   ' 30 without replacement

' Built-in tool (samples WITH replacement, or periodically):
'   Data ▸ Data Analysis ▸ Sampling  ▸ Random / Periodic

' ── Systematic sample ───────────────────────────────────────────────
=ROUNDDOWN(500/30, 0)                     ' k = interval -> 16
=RANDBETWEEN(1, 16)                       ' random start r
=INDEX($A$2:$A$501, $F$2 + (ROW()-2)*$F$1)   ' r, r+k, r+2k, …

' ── Stratified sample: proportional allocation ──────────────────────
' Stratum sizes in B2:B5, desired total n in E1
=ROUND(E1*B2/SUM($B$2:$B$5), 0)           ' n for this stratum
=COUNTIF($C$2:$C$501, A2)                 ' actual stratum size from raw data

' ── Randomly assign subjects to two treatment groups ────────────────
=IF(RAND()<0.5, "Treatment", "Control")
=CHOOSE(RANDBETWEEN(1,3), "A", "B", "Placebo")   ' three arms

---

R

set.seed(2024)
population <- data.frame(
  id     = 1:500,
  region = rep(c("North", "South", "East", "West"), times = c(200, 150, 100, 50)),
  score  = round(rnorm(500, 70, 12), 1)
)

# ── Simple random sample ───────────────────────────────────────────
srs <- population[sample(nrow(population), 30), ]
head(srs)
sample(1:500, 30, replace = FALSE)          # just the ids
sample(1:500, 30, replace = TRUE)           # with replacement

# ── Systematic sample ──────────────────────────────────────────────
n <- 30; N <- nrow(population)
k <- floor(N / n)                            # 16
start <- sample(1:k, 1)
idx <- seq(start, N, by = k)[1:n]
systematic <- population[idx, ]

# ── Stratified sample, proportional allocation ─────────────────────
library(dplyr)
strat <- population %>%
  group_by(region) %>%
  slice_sample(prop = 30 / N) %>%            # proportional
  ungroup()
table(strat$region)

# Equal allocation instead
population %>% group_by(region) %>% slice_sample(n = 8) %>% ungroup()

# ── Cluster sample: pick whole clusters ────────────────────────────
population$cluster <- rep(1:50, each = 10)
chosen <- sample(unique(population$cluster), 3)
clustered <- population[population$cluster %in% chosen, ]

# ── Random assignment to treatments ────────────────────────────────
subjects <- paste0("S", 1:60)
assign <- data.frame(subject = subjects,
                     group = sample(rep(c("Treatment", "Control"), each = 30)))
table(assign$group)

# ── Comparing the precision of SRS and stratified sampling ─────────
srs_means <- replicate(2000, mean(sample(population$score, 40)))
str_means <- replicate(2000, {
  s <- population %>% group_by(region) %>% slice_sample(prop = 40/N) %>% ungroup()
  mean(s$score)
})
c(SRS = sd(srs_means), Stratified = sd(str_means))   # stratified is usually smaller

---

Python

import numpy as np
import pandas as pd

rng = np.random.default_rng(2024)

population = pd.DataFrame({
    "id": np.arange(1, 501),
    "region": np.repeat(["North", "South", "East", "West"], [200, 150, 100, 50]),
    "score": rng.normal(70, 12, 500).round(1),
})

# ── Simple random sample ───────────────────────────────────────────
srs = population.sample(n=30, random_state=2024)
population.sample(frac=0.06, random_state=1)          # 6% of the rows
rng.choice(population["id"], 30, replace=False)       # just the ids

# ── Systematic sample ──────────────────────────────────────────────
n, N = 30, len(population)
k = N // n                                             # 16
start = rng.integers(0, k)
systematic = population.iloc[start::k].head(n)

# ── Stratified sample, proportional allocation ─────────────────────
strat = (population
         .groupby("region", group_keys=False)
         .apply(lambda g: g.sample(frac=30 / N, random_state=1)))
strat["region"].value_counts()

# Equal allocation
population.groupby("region", group_keys=False).apply(lambda g: g.sample(8))

# scikit-learn does stratified splits directly
# from sklearn.model_selection import train_test_split
# _, sample = train_test_split(population, test_size=30/N, stratify=population["region"])

# ── Cluster sample ─────────────────────────────────────────────────
population["cluster"] = np.repeat(np.arange(1, 51), 10)
chosen = rng.choice(population["cluster"].unique(), 3, replace=False)
clustered = population[population["cluster"].isin(chosen)]

# ── Random assignment to treatments ────────────────────────────────
groups = np.repeat(["Treatment", "Control"], 30)
rng.shuffle(groups)
assign = pd.DataFrame({"subject": [f"S{i}" for i in range(1, 61)], "group": groups})
assign["group"].value_counts()

# ── Comparing precision ────────────────────────────────────────────
srs_means = [population["score"].sample(40).mean() for _ in range(2000)]
np.std(srs_means, ddof=1)

---

Quick Reference

Task	Excel	R	Python
Simple random sample	SORTBY(range, RANDARRAY(n))	sample(x, n)	df.sample(n=…)
With replacement	RANDBETWEEN	sample(x, n, TRUE)	df.sample(n, replace=True)
Reproducible	(not available)	set.seed(k)	random_state=k
Systematic	INDEX with step k	seq(start, N, by = k)	df.iloc[start::k]
Stratified	Sample within each group	group_by %>% slice_sample	groupby().apply(sample)
Cluster	Sample cluster ids, keep all members	x[cluster %in% chosen, ]	df[df.cluster.isin(chosen)]
Random assignment	IF(RAND()<0.5, …)	sample(rep(groups, each=k))	rng.shuffle(groups)
Sampling tool	Data Analysis ▸ Sampling	—	—

---

Common Mistakes

• Calling a convenience sample "random" because no one was deliberately chosen.
• Confusing stratified (some from every group) with cluster (all from some groups).
• Believing a bigger sample fixes bias.
• Reporting a confidence interval from a voluntary-response poll — the formula assumes random sampling.
• Claiming causation from an observational study. Only randomized assignment licenses that.
• Systematic sampling on a list with a hidden periodic structure.

---

Exercises: 08-01: Exercises — Sampling Methods and Bias

---

⬅️ Previous: 07-02: The Normal Distribution and Z-Scores ➡️ Next: 08-02: Sampling Distributions and the Central Limit Theorem