11-02: Two-Sample t-Test (Independent Samples)¶
Comparing two separate groups — treatment vs. control, method A vs. method B, men vs. women. The samples must be independent: knowing who is in one tells you nothing about who is in the other. (If the same subjects appear in both, you need the paired test of 11-03.)
The Hypotheses¶
H₀: μ₁ = μ₂ (equivalently μ₁ − μ₂ = 0)
H₁: μ₁ ≠ μ₂ two-tailed
μ₁ > μ₂ right-tailed
μ₁ < μ₂ left-tailed
Two Versions of the Test¶
Welch's t-test — unequal variances (the default, and the safer choice)¶
x̄₁ − x̄₂
t = ──────────────────────────
_________________
/ s₁² s₂²
√ ──── + ────
n₁ n₂
( s₁²/n₁ + s₂²/n₂ )²
df = ────────────────────────────────────────── (Welch-Satterthwaite;
( s₁²/n₁ )² ( s₂²/n₂ )² usually not an integer)
─────────── + ───────────
n₁ − 1 n₂ − 1
Pooled t-test — equal variances assumed¶
(n₁ − 1)s₁² + (n₂ − 1)s₂²
Pooled variance s²_p = ──────────────────────────────
n₁ + n₂ − 2
x̄₁ − x̄₂
t = ──────────────────────────── df = n₁ + n₂ − 2
______________
/ ( 1 1 )
√ s²_p( ── + ── )
( n₁ n₂ )
Tip
Which to use? Welch's, almost always. It performs nearly as well as the pooled test when the variances are equal, and much better when they are not — so it is the default in R (t.test) and the recommendation in most modern texts. Use the pooled version when a course specifically asks for it, or when a variance test supports equality.
Testing whether the variances are equal¶
| Test | Requires | Notes |
|---|---|---|
F-test (s₁²/s₂²) |
Normality | Very sensitive to non-normality |
| Levene's test | — | Robust; the practical choice |
| Bartlett's test | Normality | More powerful when normality holds |
H₀: σ₁² = σ₂². A small p-value means the variances differ → use Welch.
Conditions¶
1. Two INDEPENDENT random samples
2. Each population approximately normal, OR both n ≥ 30
3. (Pooled version only) equal population variances
A common practical rule: if the larger sample variance is less than 4× the smaller, the pooled test is usually acceptable.
Worked Example¶
Two teaching methods are compared. Method A: n₁ = 20, x̄₁ = 78.4, s₁ = 8.2. Method B: n₂ = 18, x̄₂ = 72.1, s₂ = 9.6. Is Method A better? Test at α = 0.05.
STEP 1 H₀: μ₁ ≤ μ₂ H₁: μ₁ > μ₂ (right-tailed) α = 0.05
STEP 2 Independent samples ✓ scores approximately normal ✓
Variance check: 9.6²/8.2² = 92.16/67.24 = 1.37 < 4 → either test is fine
Pooled version
STEP 3 s²_p = [19(67.24) + 17(92.16)] / (20 + 18 − 2)
= [1277.56 + 1566.72] / 36
= 2844.28 / 36
= 79.008
SE = √(79.008 × (1/20 + 1/18)) = √(79.008 × 0.10556) = √8.3398 = 2.888
78.4 − 72.1 6.3
t = ───────────── = ─────── = 2.181 df = 36
2.888 2.888
STEP 4 Critical value: t(0.05, 36) = 1.688
2.181 > 1.688 → REJECT H₀
p-value: P(T₃₆ > 2.181) = 0.0179 ≤ 0.05 → REJECT H₀
STEP 5 There is sufficient evidence at the 5% level to conclude that
Method A produces a higher mean score than Method B.
Welch version — for comparison
SE = √(67.24/20 + 92.16/18) = √(3.362 + 5.120) = √8.482 = 2.912
t = 6.3 / 2.912 = 2.163
df = (8.482)² / [ (3.362)²/19 + (5.120)²/17 ]
= 71.94 / [ 0.5949 + 1.5421 ]
= 71.94 / 2.1370
= 33.66
p-value = P(T₃₃.₆₆ > 2.163) = 0.0189 → same conclusion
The two versions agree, as they usually do when the variances are similar.
Confidence interval for the difference (95%, Welch):
The interval excludes 0 → significant, and Method A's advantage is somewhere between about 0.4 and 12.2 points.
Effect size (Cohen's d, pooled):
Excel¶
' ══ SETUP ═══════════════════════════════════════════════════════════
' Group A in A2:A21, Group B in B2:B19
=COUNT(A2:A21) ' n1 -> 20
=AVERAGE(A2:A21) ' x̄1
=VAR.S(A2:A21) ' s1²
' ══ ONE-CLICK: ANALYSIS TOOLPAK ═════════════════════════════════════
' Data ▸ Data Analysis ▸ t-Test: Two-Sample Assuming Unequal Variances (Welch)
' Data ▸ Data Analysis ▸ t-Test: Two-Sample Assuming Equal Variances (pooled)
' Data ▸ Data Analysis ▸ F-Test Two-Sample for Variances (variance check)
' Variable 1 Range = A1:A21, Variable 2 Range = B1:B19,
' tick Labels, Hypothesized Mean Difference = 0, Alpha = 0.05
' → output: means, variances, df, t Stat, one- and two-tail p and critical values
' ══ T.TEST FUNCTION (p-value only) ══════════════════════════════════
=T.TEST(A2:A21, B2:B19, 1, 2) ' 1 tail, type 2 = pooled -> 0.01786
=T.TEST(A2:A21, B2:B19, 2, 2) ' 2 tails, pooled -> 0.03571
=T.TEST(A2:A21, B2:B19, 1, 3) ' 1 tail, type 3 = Welch -> 0.01893
' ══ BUILT FROM SUMMARY STATISTICS ═══════════════════════════════════
' D1=n1 20, D2=x̄1 78.4, D3=s1 8.2, E1=n2 18, E2=x̄2 72.1, E3=s2 9.6
' Pooled
=((D1-1)*D3^2+(E1-1)*E3^2)/(D1+E1-2) ' s²_p -> 79.0078
=SQRT(G1*(1/D1+1/E1)) ' SE -> 2.88788
=(D2-E2)/G2 ' t -> 2.18153
=D1+E1-2 ' df -> 36
=T.DIST.RT(G3, G4) ' right p -> 0.01786
=T.INV(1-0.05, G4) ' critical t -> 1.68830
' Welch
=SQRT(D3^2/D1 + E3^2/E1) ' SE -> 2.91237
=(D2-E2)/H1 ' t -> 2.16315
=(D3^2/D1+E3^2/E1)^2 / ((D3^2/D1)^2/(D1-1) + (E3^2/E1)^2/(E1-1)) ' df -> 33.66
=T.DIST.RT(H2, H3) ' right p -> 0.01893
' ══ CONFIDENCE INTERVAL FOR THE DIFFERENCE ══════════════════════════
=(D2-E2) - T.INV.2T(0.05, H3)*H1 ' lower -> 0.3773
=(D2-E2) + T.INV.2T(0.05, H3)*H1 ' upper -> 12.2227
' ══ VARIANCE CHECK (F-test) ═════════════════════════════════════════
=MAX(D3^2,E3^2)/MIN(D3^2,E3^2) ' F ratio -> 1.3706
=F.TEST(A2:A21, B2:B19) ' two-tailed p-value
=F.INV.RT(0.025, E1-1, D1-1) ' critical F
' ══ EFFECT SIZE ═════════════════════════════════════════════════════
=(D2-E2)/SQRT(G1) ' Cohen's d -> 0.7088
R¶
set.seed(5)
A <- round(rnorm(20, 78.4, 8.2), 1)
B <- round(rnorm(18, 72.1, 9.6), 1)
# ══ THE TEST ════════════════════════════════════════════════════════
t.test(A, B) # Welch — R's DEFAULT
t.test(A, B, var.equal = TRUE) # pooled
t.test(A, B, alternative = "greater") # one-sided Welch
# Formula interface — needs LONG format
scores <- data.frame(score = c(A, B),
method = rep(c("A", "B"), c(length(A), length(B))))
t.test(score ~ method, data = scores)
t.test(score ~ method, data = scores, var.equal = TRUE)
# ══ FROM SUMMARY STATISTICS ═════════════════════════════════════════
two_sample_t <- function(m1, s1, n1, m2, s2, n2, pooled = FALSE) {
if (pooled) {
sp2 <- ((n1-1)*s1^2 + (n2-1)*s2^2) / (n1 + n2 - 2)
se <- sqrt(sp2 * (1/n1 + 1/n2))
df <- n1 + n2 - 2
} else {
se <- sqrt(s1^2/n1 + s2^2/n2)
df <- (s1^2/n1 + s2^2/n2)^2 /
((s1^2/n1)^2/(n1-1) + (s2^2/n2)^2/(n2-1))
}
t <- (m1 - m2) / se
list(se = se, df = df, t = t,
p_two = 2 * pt(abs(t), df, lower.tail = FALSE),
p_right = pt(t, df, lower.tail = FALSE),
ci = (m1 - m2) + c(-1, 1) * qt(0.975, df) * se)
}
two_sample_t(78.4, 8.2, 20, 72.1, 9.6, 18, pooled = TRUE)
# $se 2.8879 $df 36 $t 2.1815 $p_right 0.01786 $ci 0.4416 12.1584
two_sample_t(78.4, 8.2, 20, 72.1, 9.6, 18)
# $se 2.9124 $df 33.66 $t 2.1631 $p_right 0.01893 $ci 0.3773 12.2227
# ══ CHECKING ASSUMPTIONS ════════════════════════════════════════════
boxplot(score ~ method, data = scores, col = c("#5B2A86", "#0FA3A3"))
shapiro.test(A); shapiro.test(B) # normality of each group
var.test(A, B) # F-test for equal variances
library(car); leveneTest(score ~ factor(method), data = scores) # robust
bartlett.test(score ~ method, data = scores)
# ══ EFFECT SIZE ═════════════════════════════════════════════════════
sp <- sqrt(((length(A)-1)*var(A) + (length(B)-1)*var(B)) /
(length(A) + length(B) - 2))
(mean(A) - mean(B)) / sp # Cohen's d
# library(effectsize); cohens_d(score ~ method, data = scores)
# ══ NONPARAMETRIC ALTERNATIVE ═══════════════════════════════════════
wilcox.test(A, B) # Mann-Whitney U
Python¶
import numpy as np
import pandas as pd
from scipy import stats
rng = np.random.default_rng(5)
A = rng.normal(78.4, 8.2, 20).round(1)
B = rng.normal(72.1, 9.6, 18).round(1)
# ══ THE TEST ════════════════════════════════════════════════════════
stats.ttest_ind(A, B, equal_var=False) # Welch — RECOMMENDED
stats.ttest_ind(A, B, equal_var=True) # pooled
stats.ttest_ind(A, B, equal_var=False, alternative="greater") # one-sided
res = stats.ttest_ind(A, B, equal_var=False)
res.statistic, res.pvalue, res.df
res.confidence_interval(confidence_level=0.95)
# ══ FROM SUMMARY STATISTICS ═════════════════════════════════════════
stats.ttest_ind_from_stats(mean1=78.4, std1=8.2, nobs1=20,
mean2=72.1, std2=9.6, nobs2=18,
equal_var=True) # pooled: t=2.1815, p=0.0357
stats.ttest_ind_from_stats(78.4, 8.2, 20, 72.1, 9.6, 18, equal_var=False)
# Welch: t=2.1631, p=0.0379
def two_sample_t(m1, s1, n1, m2, s2, n2, pooled=False):
if pooled:
sp2 = ((n1-1)*s1**2 + (n2-1)*s2**2) / (n1 + n2 - 2)
se = np.sqrt(sp2 * (1/n1 + 1/n2)); df = n1 + n2 - 2
else:
se = np.sqrt(s1**2/n1 + s2**2/n2)
df = (s1**2/n1 + s2**2/n2)**2 / \
((s1**2/n1)**2/(n1-1) + (s2**2/n2)**2/(n2-1))
t = (m1 - m2) / se
tc = stats.t.ppf(0.975, df)
return {"se": se, "df": df, "t": t,
"p_two": 2 * stats.t.sf(abs(t), df),
"p_right": stats.t.sf(t, df),
"ci": ((m1-m2) - tc*se, (m1-m2) + tc*se)}
two_sample_t(78.4, 8.2, 20, 72.1, 9.6, 18, pooled=True)
# ══ CHECKING ASSUMPTIONS ════════════════════════════════════════════
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.boxplot([A, B], tick_labels=["A", "B"])
stats.shapiro(A), stats.shapiro(B) # normality
stats.levene(A, B) # robust equal-variance test
stats.bartlett(A, B) # assumes normality
# ══ EFFECT SIZE ═════════════════════════════════════════════════════
n1, n2 = A.size, B.size
sp = np.sqrt(((n1-1)*A.var(ddof=1) + (n2-1)*B.var(ddof=1)) / (n1 + n2 - 2))
(A.mean() - B.mean()) / sp # Cohen's d
# ══ NONPARAMETRIC ALTERNATIVE ═══════════════════════════════════════
stats.mannwhitneyu(A, B)
Quick Reference¶
| Task | Excel | R | Python |
|---|---|---|---|
| Welch t-test | ToolPak ▸ Unequal Variances; T.TEST(...,3) |
t.test(x, y) |
ttest_ind(x, y, equal_var=False) |
| Pooled t-test | ToolPak ▸ Equal Variances; T.TEST(...,2) |
t.test(x, y, var.equal=TRUE) |
ttest_ind(x, y, equal_var=True) |
| From summary stats | manual formulas | user function | ttest_ind_from_stats(...) |
| One-sided | T.TEST(..., 1, type) |
alternative = "greater" |
alternative="greater" |
| Equal-variance check | ToolPak ▸ F-Test; F.TEST |
var.test, car::leveneTest |
stats.levene, stats.bartlett |
| Normality check | Q-Q scatter | shapiro.test |
stats.shapiro |
| CI for the difference | manual formula | t.test(x,y)$conf.int |
res.confidence_interval() |
| Effect size | (x̄1-x̄2)/SQRT(sp2) |
effectsize::cohens_d |
manual |
| Nonparametric | — | wilcox.test(x, y) |
stats.mannwhitneyu(x, y) |
Common Mistakes¶
- Using an independent two-sample test on paired data (before/after on the same subjects). That throws away the pairing and loses power — see 11-03.
- Assuming equal variances without checking, then reporting the pooled test.
- Excel's
T.TESTtypeargument: 1 = paired, 2 = pooled, 3 = Welch. Passing 2 when the data is paired is a silent error. - Getting the sign backwards by swapping which group is "1". Decide first, keep it consistent through the CI.
- Reporting the Welch
dfas a whole number without noting it is an approximation. - Ignoring that a two-tailed p-value is double the one-tailed one; Excel's ToolPak prints both, so use the right row.
Exercises: 11-02: Exercises — Two-Sample t-Test
⬅️ Previous: 11-01: One-Sample t-Test ➡️ Next: 11-03: Paired t-Test