08-01: Exercises — Sampling Methods and Bias

Notes reference: 08-01: Sampling Methods and Bias

---

Q1: Identify the sampling method

Name the method used in each scenario.

1. A researcher numbers all 4,000 employees and uses a random number generator to pick 200.
2. A quality inspector tests every 50th item off the line, starting from a randomly chosen one of the first 50.
3. A pollster divides voters into four age bands and randomly samples 100 from each.
4. A school district randomly selects 6 of its 40 schools and surveys every teacher in those 6.
5. A journalist interviews shoppers outside one supermarket on a Tuesday morning.
6. A study randomly picks 5 states, then 3 districts within each, then 50 households within each district.

Solution

1. SIMPLE RANDOM SAMPLING
2. SYSTEMATIC SAMPLING          (k = 50, random start)
3. STRATIFIED SAMPLING          (some from EVERY age band)
4. CLUSTER SAMPLING             (ALL from SOME schools)
5. CONVENIENCE SAMPLING         (non-probability — no inference is valid)
6. MULTISTAGE SAMPLING          (nested random selection)

---

Q2: Stratified vs. cluster

Both split the population into groups. Complete the table, then design one of each for surveying university students about campus food.

Solution

	Stratified	Cluster
Groups are	Homogeneous within, different between	Heterogeneous within, similar between
You sample	Some from every group	All from some groups
Main goal	Precision	Cost / logistics
Precision vs. SRS	Usually better	Usually worse

STRATIFIED DESIGN
  Strata: year of study (1st, 2nd, 3rd, 4th) — students within a year
  have similar campus habits, and the years differ from each other.
  Randomly sample 75 students from each year → n = 300.
  Ensures every year is represented; reduces the standard error.

CLUSTER DESIGN
  Clusters: the 60 residence halls — each hall contains a MIX of years
  and majors, so halls resemble one another.
  Randomly select 5 halls and survey every resident → n ≈ 300.
  Far cheaper (5 locations, not 300 individuals), less precise.

---

Q3: Sampling error or bias?

Classify each, and say whether a bigger sample would help.

1. A poll of 1,000 voters estimates 52% support; the true figure is 50%.
2. An online survey is answered mostly by people with strong opinions.
3. A telephone survey uses only landlines.
4. Two different random samples of 500 give 48% and 53%.
5. A satisfaction survey is handed out only to customers who complete a purchase.

Solution

1. SAMPLING ERROR   — random variation. A bigger sample SHRINKS it.
2. BIAS (nonresponse / voluntary response). Bigger sample does NOT help.
3. BIAS (coverage / undercoverage). Bigger sample does NOT help.
4. SAMPLING ERROR   — the expected spread between independent samples.
5. BIAS (selection). Unhappy customers who abandoned the purchase are
   never surveyed. Bigger sample does NOT help.

THE KEY DISTINCTION:
   Sampling error is random, measurable, and shrinks with √n.
   Bias is systematic, invisible in the data, and a bigger sample only
   makes the wrong answer more precise.

---

Q4: Diagnose the bias

Name the specific bias and propose a fix.

1. A magazine mails a survey to subscribers; 4% respond.
2. A researcher rates the "top 10 mutual funds" using only funds still operating today.
3. A survey asks "Do you support the reckless proposal to cut school funding?"
4. A health study recruits volunteers at a marathon.
5. An employee survey is administered by each worker's direct manager.

Solution

1. NONRESPONSE BIAS. A 4% response rate means 96% are unrepresented, and
   responders differ systematically from non-responders.
   FIX: follow-up contacts, incentives, and a nonresponse analysis
        comparing responders to known population characteristics.

2. SURVIVORSHIP BIAS. Funds that failed have been deleted, so the
   surviving set looks artificially strong.
   FIX: use a survivorship-bias-free database including closed funds.

3. QUESTION-WORDING (leading question) BIAS. "Reckless" pre-loads the answer.
   FIX: neutral wording — "Do you support or oppose the proposed change
        to school funding?" and randomize the order of options.

4. SELECTION BIAS. Marathon runners are far healthier than average.
   FIX: sample from a general population frame, not a self-selected venue.

5. RESPONSE BIAS (social desirability / fear of reprisal). Employees will
   not criticise a manager who can see their answers.
   FIX: anonymous collection through a neutral third party.

---

Q5: Observational study or experiment?

Classify each and say whether a causal claim is justified.

1. Researchers randomly assign 200 patients to a new drug or a placebo.
2. Researchers compare cancer rates between smokers and non-smokers.
3. A company randomly shows half its website visitors a new checkout page.
4. A study finds that students who eat breakfast score higher on tests.

Solution

1. EXPERIMENT (randomized, controlled).  CAUSAL claim JUSTIFIED.

2. OBSERVATIONAL.  Association only. Possible confounders: diet, exercise,
   income, occupation. (The smoking–cancer causal case was eventually made
   through dose-response, biological mechanism, and consistency across
   dozens of studies — not from a single observational comparison.)

3. EXPERIMENT (an A/B test is a randomized controlled trial).
   CAUSAL claim JUSTIFIED for the population of site visitors.

4. OBSERVATIONAL.  Association only. Confounders: household income,
   sleep, parental involvement, general health. Eating breakfast may
   be a MARKER for a stable home rather than a CAUSE of higher scores.

---

Q6: Design an experiment

A company wants to test whether a new training programme improves productivity. 120 employees are available.

Apply the four principles of experimental design.

Solution

CONTROL
  Two groups: 60 receive the new programme, 60 continue with the existing
  one (an active control, not "nothing" — otherwise you measure attention,
  not training). Keep workload, tools, and measurement period identical.

RANDOMIZATION
  Assign employees to groups AT RANDOM. This balances unmeasured
  confounders (experience, motivation, shift) on average.
     R:      sample(rep(c("New","Existing"), each = 60))
     Excel:  =IF(RAND()<0.5,"New","Existing")  then rebalance to 60/60

REPLICATION
  60 per group gives enough power to detect a moderate effect.
  Check with power.t.test(delta = ..., sd = ..., power = 0.80).

BLINDING
  Employees will know which training they attended, but the SUPERVISORS
  who score productivity should not. Better still, use an objective
  automated metric (units completed, error rate).

BLOCKING (optional refinement)
  If experience matters, block on tenure (<2 yr, 2–5 yr, >5 yr) and
  randomize WITHIN each block. This is the experimental analogue of
  stratified sampling and increases precision.

ANALYSIS
  Independent two-sample t-test (11-02), reporting the mean difference,
  a 95% confidence interval, and Cohen's d.

---

Q7: Draw the samples in software

From a 500-row population with a region column, draw (a) an SRS of 40, (b) a systematic sample of 40, (c) a proportionally stratified sample of 40.

Solution

' (a) SIMPLE RANDOM SAMPLE
'     Add a helper column  =RAND()  , sort the WHOLE block by it, take 40 rows
=INDEX(SORTBY(A2:D501, RANDARRAY(500)), SEQUENCE(40), {1,2,3,4})
'     Or:  Data ▸ Data Analysis ▸ Sampling ▸ Random (samples WITH replacement)

' (b) SYSTEMATIC SAMPLE
=ROUNDDOWN(500/40, 0)                       ' k = 12
=RANDBETWEEN(1, 12)                         ' random start r
=INDEX($A$2:$A$501, $G$2 + (ROW()-2)*$G$1)  ' r, r+k, r+2k, …

' (c) STRATIFIED, PROPORTIONAL
'     Stratum sizes in B2:B5, total n in E1
=ROUND($E$1*B2/SUM($B$2:$B$5), 0)           ' how many from this stratum
'     Then run (a) separately within each stratum's rows.

set.seed(1)
# (a) SRS
srs <- pop[sample(nrow(pop), 40), ]

# (b) systematic
k <- floor(nrow(pop) / 40)
start <- sample(k, 1)
sys <- pop[seq(start, nrow(pop), by = k)[1:40], ]

# (c) stratified, proportional
library(dplyr)
strat <- pop %>% group_by(region) %>%
  slice_sample(prop = 40 / nrow(pop)) %>% ungroup()
table(strat$region)

rng = np.random.default_rng(1)
srs = pop.sample(n=40, random_state=1)                                  # (a)

k = len(pop) // 40
start = rng.integers(0, k)
sys = pop.iloc[start::k].head(40)                                       # (b)

strat = (pop.groupby("region", group_keys=False)
            .apply(lambda g: g.sample(frac=40/len(pop), random_state=1)))  # (c)
strat["region"].value_counts()

---

Q8: Does stratifying actually help?

Demonstrate, by simulation, that a stratified sample gives a smaller standard error than an SRS of the same size when the strata really differ.

Solution

set.seed(7)
# A population where region genuinely matters
pop <- data.frame(
  region = rep(c("N", "S", "E", "W"), times = c(200, 150, 100, 50)),
  score  = c(rnorm(200, 60, 8), rnorm(150, 75, 8),
             rnorm(100, 85, 8), rnorm( 50, 95, 8))
)
true_mean <- mean(pop$score)

srs_means <- replicate(2000, mean(sample(pop$score, 40)))

library(dplyr)
str_means <- replicate(2000, {
  s <- pop %>% group_by(region) %>% slice_sample(prop = 40/500) %>% ungroup()
  mean(s$score)
})

round(c(true          = true_mean,
        srs_mean      = mean(srs_means),
        strat_mean    = mean(str_means),
        srs_se        = sd(srs_means),
        strat_se      = sd(str_means)), 3)

TYPICAL OUTPUT

    true      srs_mean   strat_mean   srs_se   strat_se
   73.0        73.0        73.0        2.20      1.27

BOTH methods are UNBIASED — each centres on the true mean of 73.
STRATIFIED has a smaller standard error (1.27 vs. 2.20), so the same
budget buys an estimate roughly 1.7× more precise.

WHY: stratifying removes the between-region variation from the sampling
error entirely. Every sample is guaranteed the right regional mix, so
the estimate cannot be thrown off by an unlucky draw of mostly-West rows.

WHEN IT DOESN'T HELP: if the strata have nearly identical means,
stratifying costs effort and buys almost nothing.

---

⬅️ Previous: 07-02: Exercises — The Normal Distribution and Z-Scores ➡️ Next: 08-02: Exercises — Sampling Distributions and the Central Limit Theorem