10-01: Hypothesis Testing Fundamentals¶
A hypothesis test answers a yes/no question about a population using sample evidence: is this difference real, or could chance alone have produced it? Every test in Chapters 10–13 follows the same five steps — learn the framework once and the rest is choosing a formula.
The Two Hypotheses¶
H₀ Null hypothesis The status quo, "no effect", "no difference".
ALWAYS contains an equality: =, ≤, or ≥
This is what you assume TRUE while testing.
H₁ Alternative hypothesis The research claim, what you hope to show.
(or Hₐ) ALWAYS strictly inequality: ≠, <, or >
They must be mutually exclusive and exhaustive, and the parameter (μ, p, σ) never carries a sample statistic — you never write H₀: x̄ = 50.
The three forms¶
| Test type | H₀ |
H₁ |
Rejection region |
|---|---|---|---|
| Two-tailed | μ = μ₀ |
μ ≠ μ₀ |
Both tails, α/2 each |
| Right-tailed | μ ≤ μ₀ |
μ > μ₀ |
Right tail, α |
| Left-tailed | μ ≥ μ₀ |
μ < μ₀ |
Left tail, α |
Translating words into hypotheses:
| Claim in words | H₁ |
|---|---|
| "is different from", "has changed" | ≠ (two-tailed) |
| "is more than", "exceeds", "is greater than", "improved" | > (right-tailed) |
| "is less than", "is below", "reduced", "is faster" | < (left-tailed) |
| "is at least" | H₀: μ ≥ … → H₁: μ < … |
| "is at most" | H₀: μ ≤ … → H₁: μ > … |
Tip
The direction of H₁ is the direction of the research claim. Set H₁ first, then H₀ is whatever is left over.
Two Types of Error¶
REALITY
H₀ true H₀ false
┌─────────────┬─────────────┐
Reject H₀ │ Type I (α) │ Correct │ ← power = 1 − β
├─────────────┼─────────────┤
Fail to reject│ Correct │ Type II (β) │
└─────────────┴─────────────┘
| Definition | Analogy (a criminal trial, H₀ = innocent) |
|
|---|---|---|
Type I error α |
Rejecting a true H₀ — a false positive |
Convicting an innocent person |
Type II error β |
Failing to reject a false H₀ — a false negative |
Acquitting a guilty person |
Power 1 − β |
Correctly rejecting a false H₀ |
Convicting a guilty person |
Trade-off: lowering α (being more cautious about false alarms) raises β. The only way to reduce both is to increase the sample size.
Power increases with: larger n, larger true effect size, smaller σ, larger α.
The Significance Level α¶
α is the Type I error rate you are willing to accept, chosen before looking at the data.
α |
Used when |
|---|---|
| 0.10 | Exploratory work, low cost of a false positive |
| 0.05 | The default in most fields |
| 0.01 | Higher stakes — medical, safety-critical |
| 0.001 | Very high stakes; also common with many simultaneous tests |
The Five Steps¶
1. STATE H₀ and H₁, and choose α.
2. CHECK Conditions: random sample, independence, normality / CLT,
sample size rules.
3. COMPUTE The test statistic:
(sample statistic) − (hypothesized value)
test stat = ───────────────────────────────────────────────
standard error of the statistic
4. DECIDE Either route gives the SAME answer:
Critical-value route: is the test statistic in the
rejection region?
p-value route: is p ≤ α ?
5. CONCLUDE In the CONTEXT of the problem, in plain language.
The p-Value¶
The p-value is the probability of getting a test statistic AT LEAST AS
EXTREME as the one observed, ASSUMING H₀ is true.
Two-tailed: p = 2 × P(Z > |z_observed|)
Right-tailed: p = P(Z > z_observed)
Left-tailed: p = P(Z < z_observed)
The decision rule, in five words:
p ≤ α → REJECT H₀ "the result is statistically significant"
p > α → FAIL TO REJECT H₀ "insufficient evidence"
Warning
What a p-value is NOT:
- Not the probability that
H₀is true. - Not the probability that the result was due to chance.
- Not a measure of effect size — a tiny, meaningless difference becomes "significant" with a large enough
n. p = 0.049andp = 0.051are not meaningfully different, despite landing on opposite sides of 0.05.
Wording the Conclusion¶
You never "accept" H₀ — absence of evidence is not evidence of absence.
| Decision | Correct wording |
|---|---|
Reject H₀ |
"There is sufficient evidence at the 5% level to conclude that the mean commute time exceeds 30 minutes." |
Fail to reject H₀ |
"There is not sufficient evidence at the 5% level to conclude that the mean commute time exceeds 30 minutes." |
Statistical vs. Practical Significance¶
A new teaching method raises test scores by 0.4 points with n = 50,000 and p < 0.001. Statistically significant — and practically worthless.
Always report an effect size and a confidence interval alongside the p-value:
Confidence Intervals and Tests Agree¶
For a two-tailed test at level α:
If the (1 − α) confidence interval EXCLUDES the hypothesized value
→ reject H₀ at level α.
If it INCLUDES the hypothesized value
→ fail to reject H₀.
A 95% CI of (23.97, 30.83) for mean commute time means H₀: μ = 30 is not rejected at α = 0.05, while H₀: μ = 22 is. The interval carries more information than the p-value, which is why journals increasingly require it.
Worked Example — Setting Up a Test¶
A manufacturer claims its batteries last at least 40 hours. A consumer group suspects they last less and tests 36 batteries, finding x̄ = 38.4 hours with σ = 4.2 hours. Test at α = 0.05.
STEP 1 H₀: μ ≥ 40 (the manufacturer's claim)
H₁: μ < 40 (the consumer group's suspicion — LEFT-tailed)
α = 0.05
STEP 2 Random sample assumed; n = 36 ≥ 30 → CLT applies; σ known → z-test
STEP 3 SE = 4.2 / √36 = 0.70
z = (38.4 − 40) / 0.70 = −2.286
STEP 4 Critical value route: z_critical = −1.645 (left tail, α = 0.05)
−2.286 < −1.645 → REJECT H₀
p-value route: p = P(Z < −2.286) = 0.0111
0.0111 ≤ 0.05 → REJECT H₀
STEP 5 There is sufficient evidence at the 5% level to conclude that the
mean battery life is less than 40 hours.
The full mechanics of this test are in 10-02.
Critical Values You Will Reuse¶
α |
Left-tailed z |
Right-tailed z |
Two-tailed ±z |
|---|---|---|---|
| 0.10 | −1.282 | +1.282 | ±1.645 |
| 0.05 | −1.645 | +1.645 | ±1.960 |
| 0.01 | −2.326 | +2.326 | ±2.576 |
Excel¶
' ── Critical values ─────────────────────────────────────────────────
=NORM.S.INV(0.05) ' left-tailed α=0.05 -> -1.64485
=NORM.S.INV(0.95) ' right-tailed α=0.05 -> +1.64485
=NORM.S.INV(1-0.05/2) ' two-tailed α=0.05 -> ±1.95996
=ABS(NORM.S.INV(0.05/2)) ' same
=T.INV(0.05, 14) ' left-tailed t, df=14 -> -1.76131
=T.INV.2T(0.05, 14) ' two-tailed t -> ±2.14479
' ── p-values from a test statistic ──────────────────────────────────
=NORM.S.DIST(-2.286, TRUE) ' left-tailed p -> 0.011127
=1-NORM.S.DIST(2.286, TRUE) ' right-tailed p -> 0.011127
=2*(1-NORM.S.DIST(ABS(-2.286),TRUE))' two-tailed p -> 0.022254
=T.DIST(-2.286, 14, TRUE) ' left-tailed t p-value
=T.DIST.RT(2.286, 14) ' right-tailed t p-value
=T.DIST.2T(ABS(-2.286), 14) ' two-tailed t p-value
' ── Decision helper ─────────────────────────────────────────────────
=IF(B10<=B11, "Reject H0", "Fail to reject H0") ' B10 = p, B11 = alpha
' ── Effect size ─────────────────────────────────────────────────────
=(AVERAGE(A2:A37)-40)/STDEV.S(A2:A37) ' Cohen's d
' ── Power / sample size ─────────────────────────────────────────────
' Excel has no built-in power function. For a one-sample z-test:
=1-NORM.S.DIST(NORM.S.INV(1-B11) - (B12-B13)/(B14/SQRT(B15)), TRUE)
' B11=alpha, B12=true mean, B13=H0 mean, B14=sigma, B15=n
R¶
# ── Critical values ────────────────────────────────────────────────
qnorm(0.05) # left-tailed z -> -1.644854
qnorm(0.95) # right-tailed z -> +1.644854
qnorm(1 - 0.05/2) # two-tailed z -> +1.959964
qt(0.05, df = 14) # left-tailed t -> -1.761310
qt(1 - 0.05/2, 14) # two-tailed t -> +2.144787
# ── p-values from a test statistic ─────────────────────────────────
z <- -2.286
pnorm(z) # left-tailed -> 0.011127
pnorm(abs(z), lower.tail = FALSE) # right-tailed -> 0.011127
2 * pnorm(abs(z), lower.tail = FALSE)# two-tailed -> 0.022254
pt(z, df = 14) # t versions
2 * pt(abs(z), 14, lower.tail = FALSE)
# ── A reusable decision reporter ───────────────────────────────────
decide <- function(p, alpha = 0.05)
if (p <= alpha) "Reject H0 — statistically significant"
else "Fail to reject H0 — insufficient evidence"
decide(0.0111) # "Reject H0 ..."
# ── Effect size ────────────────────────────────────────────────────
cohens_d <- function(x, mu0) (mean(x) - mu0) / sd(x)
# library(effectsize); cohens_d(x, mu = 40)
# ── Power and sample size (built in) ───────────────────────────────
power.t.test(n = 36, delta = 1.6, sd = 4.2,
sig.level = 0.05, type = "one.sample", alternative = "one.sided")
# power ≈ 0.64
power.t.test(delta = 1.6, sd = 4.2, sig.level = 0.05, power = 0.80,
type = "one.sample", alternative = "one.sided")$n
# n ≈ 55 needed for 80% power
# ── Seeing what alpha means: simulate under a TRUE H0 ──────────────
set.seed(11)
pvals <- replicate(10000, t.test(rnorm(30, mean = 100, sd = 15), mu = 100)$p.value)
mean(pvals <= 0.05) # ≈ 0.05 — the Type I error rate, by construction
hist(pvals, breaks = 20, col = "#8A5FBF", border = NA,
main = "p-values when H0 is TRUE", xlab = "p") # flat / uniform
Python¶
import numpy as np
from scipy import stats
from statsmodels.stats.power import TTestPower
# ── Critical values ────────────────────────────────────────────────
stats.norm.ppf(0.05) # left-tailed z -> -1.644854
stats.norm.ppf(0.95) # right-tailed z -> +1.644854
stats.norm.ppf(1 - 0.05/2) # two-tailed z -> +1.959964
stats.t.ppf(0.05, df=14) # left-tailed t -> -1.761310
stats.t.ppf(1 - 0.05/2, 14) # two-tailed t -> +2.144787
# ── p-values from a test statistic ─────────────────────────────────
z = -2.286
stats.norm.cdf(z) # left-tailed -> 0.011127
stats.norm.sf(abs(z)) # right-tailed -> 0.011127
2 * stats.norm.sf(abs(z)) # two-tailed -> 0.022254
stats.t.cdf(z, df=14)
2 * stats.t.sf(abs(z), 14)
# ── A reusable decision reporter ───────────────────────────────────
def decide(p, alpha=0.05):
return ("Reject H0 — statistically significant" if p <= alpha
else "Fail to reject H0 — insufficient evidence")
decide(0.0111)
# ── Effect size ────────────────────────────────────────────────────
def cohens_d(x, mu0):
x = np.asarray(x)
return (x.mean() - mu0) / x.std(ddof=1)
# ── Power and sample size ──────────────────────────────────────────
analysis = TTestPower()
analysis.power(effect_size=1.6/4.2, nobs=36, alpha=0.05, alternative="smaller")
analysis.solve_power(effect_size=1.6/4.2, power=0.80, alpha=0.05,
alternative="smaller") # required n
# ── Simulating the Type I error rate ───────────────────────────────
rng = np.random.default_rng(11)
pvals = np.array([stats.ttest_1samp(rng.normal(100, 15, 30), 100).pvalue
for _ in range(10_000)])
(pvals <= 0.05).mean() # ≈ 0.05
Quick Reference¶
| Task | Excel | R | Python |
|---|---|---|---|
| Critical z (left) | NORM.S.INV(α) |
qnorm(α) |
norm.ppf(α) |
| Critical z (two-tail) | NORM.S.INV(1-α/2) |
qnorm(1-α/2) |
norm.ppf(1-α/2) |
| Critical t (two-tail) | T.INV.2T(α, df) |
qt(1-α/2, df) |
t.ppf(1-α/2, df) |
| p-value, left | NORM.S.DIST(z,TRUE) |
pnorm(z) |
norm.cdf(z) |
| p-value, right | 1-NORM.S.DIST(z,TRUE) |
pnorm(z, lower.tail=FALSE) |
norm.sf(z) |
| p-value, two-tail | 2*(1-NORM.S.DIST(ABS(z),TRUE)) |
2*pnorm(abs(z), lower.tail=FALSE) |
2*norm.sf(abs(z)) |
| t p-value, two-tail | T.DIST.2T(ABS(t), df) |
2*pt(abs(t), df, lower.tail=FALSE) |
2*t.sf(abs(t), df) |
| Power | manual formula | power.t.test() |
TTestPower().power() |
Common Mistakes¶
- Putting the research claim in
H₀. The claim you want to support goes inH₁. - Writing hypotheses about a statistic (
x̄,p̂) instead of a parameter (μ,p). - Choosing
αafter seeing the p-value, or picking a one-tailed test after seeing which way the data went. - Saying "we accept
H₀" or "we provedH₀is true". - Reading the p-value as the probability that
H₀is true. - Confusing statistical significance with practical importance.
- Comparing a negative test statistic against a positive critical value without matching the tail.
Exercises: 10-01: Exercises — Hypothesis Testing Fundamentals
⬅️ Previous: 09-02: Confidence Interval for a Proportion ➡️ Next: 10-02: One-Sample Z-Tests