05-03: Bayes' Theorem and Counting Rules¶
Two topics that finish the probability toolkit: Bayes' theorem, which reverses a conditional probability, and the counting rules, which supply the numerator and denominator when a sample space is too large to list.
Part 1 — Bayes' Theorem¶
The formula¶
P(B | A) · P(A)
P(A | B) = ─────────────────────
P(B)
where, by the law of total probability,
P(B) = P(B | A)·P(A) + P(B | A')·P(A')
For a partition A₁, A₂, …, Aₖ:
Vocabulary¶
| Term | Meaning |
|---|---|
Prior P(A) |
What you believed before seeing the evidence |
Likelihood P(B \| A) |
How probable the evidence is if A is true |
Evidence / marginal P(B) |
How probable the evidence is overall |
Posterior P(A \| B) |
Updated belief after seeing the evidence |
Bayes' theorem is the formal answer to "the test came back positive — now what should I believe?"
Worked Example — Medical Screening¶
A disease affects 1% of a population. A screening test is:
- 95% sensitive —
P(positive | disease) = 0.95 - 90% specific —
P(negative | no disease) = 0.90, so the false-positive rate is 0.10
A randomly chosen person tests positive. What is the probability they actually have the disease?
Step 1 — write down what you know.
P(D) = 0.01 prior
P(D') = 0.99
P(+ | D) = 0.95 sensitivity
P(+ | D') = 0.10 false-positive rate (1 − specificity)
Step 2 — total probability of a positive test.
Step 3 — Bayes.
Only 8.8%. Despite a "95% accurate" test, fewer than one positive in eleven is a true case.
Why the answer feels wrong — the natural-frequency view¶
Take 10,000 people:
Disease (100) No disease (9,900)
Test positive 95 990 → 1,085 positives
Test negative 5 8,910 → 8,915 negatives
Of the 1,085 positive tests, only 95 are real: 95 / 1085 = 0.0876. The false positives come from a much larger group, so they swamp the true positives. This is base-rate neglect, and rewriting the problem as whole-number counts is the reliable cure.
Warning
P(disease | positive) and P(positive | disease) are completely different quantities. Confusing them is the prosecutor's fallacy, and it has produced real wrongful convictions.
A second example — quality control¶
From the tree in 05-02: machine A makes 60% of parts with a 2% defect rate; machine B makes 40% with a 5% defect rate. A part is defective — which machine made it?
P(defective) = (0.60)(0.02) + (0.40)(0.05) = 0.012 + 0.020 = 0.032
P(A | defective) = 0.012 / 0.032 = 0.375
P(B | defective) = 0.020 / 0.032 = 0.625
Machine A produces more parts overall, yet most defects come from B.
Part 2 — Counting Rules¶
When outcomes are equally likely, P(E) = n(E)/n(S) — so the whole problem becomes counting.
The fundamental counting principle¶
If there are n₁ ways to do the first task, n₂ the second, …, nₖ the last:
Example. A menu with 4 starters, 6 mains, 3 desserts: 4 × 6 × 3 = 72 meals.
Factorials¶
n! counts the arrangements of n distinct objects in a row.
Permutations — order matters¶
Distinguishable permutations (with repeated items):
n!
────────────────────── where k₁ + k₂ + … = n
k₁! · k₂! · … · kₚ!
Combinations — order does not matter¶
Which one?¶
| Question | Rule |
|---|---|
| "In how many orders…" / ranked positions / passwords | Permutation |
| "How many committees / groups / hands / samples…" | Combination |
| "President, secretary, treasurer from 10 people" | Permutation: 10P3 = 720 |
| "A committee of 3 from 10 people" | Combination: 10C3 = 120 |
Tip
Ask: if I swap two of the chosen items, is it a different outcome? Yes → permutation. No → combination.
Useful identities¶
nC0 = nCn = 1 nC1 = n nCr = nC(n−r)
Σ over r of nCr = 2ⁿ (the number of subsets of an n-element set)
Worked Counting Examples¶
1. A committee. From 8 women and 6 men, choose a committee of 5. How many contain exactly 3 women?
Choose 3 women from 8: 8C3 = 56
Choose 2 men from 6: 6C2 = 15
Total (product rule): 56 × 15 = 840
P(exactly 3 women) = 840 / 14C5 = 840 / 2002 = 0.4196
2. A lottery. Choose 6 numbers from 49, order irrelevant.
3. Poker. Probability of a flush (5 cards of one suit) in a 5-card hand.
Total hands = 52C5 = 2,598,960
Flushes (any suit) = 4 × 13C5 = 4 × 1287 = 5,148
P(flush, incl. straight flushes) = 5148 / 2598960 = 0.00198
4. Arrangements with repeats. How many distinct arrangements of the letters in STATISTICS?
10 letters: S×3, T×3, A×1, I×2, C×1
10! 3,628,800
─────────────── = ─────────── = 50,400
3! · 3! · 1! · 2! · 1! 72
Excel¶
' ── Bayes' theorem ──────────────────────────────────────────────────
' B1 = P(D)=0.01, B2 = P(+|D)=0.95, B3 = P(+|D')=0.10
=B2*B1 + B3*(1-B1) ' P(+) -> 0.1085
=B2*B1 / (B2*B1 + B3*(1-B1)) ' P(D|+) -> 0.0876
=(1-B2)*B1 / ((1-B2)*B1 + (1-B3)*(1-B1)) ' P(D|−) -> 0.00056
' Natural-frequency layout (build this table — it is the best teaching aid)
=10000*B1 ' with disease -> 100
=10000*B1*B2 ' true positives -> 95
=10000*(1-B1)*B3 ' false positives -> 990
=B7/(B7+B8) ' P(D|+) from counts -> 0.0876
' ── Counting rules ──────────────────────────────────────────────────
=FACT(5) ' 5! -> 120
=PERMUT(10, 3) ' 10P3 -> 720 (order matters)
=COMBIN(10, 3) ' 10C3 -> 120 (order does not)
=PERMUTATIONA(10, 3) ' with repetition allowed -> 1000
=COMBINA(10, 3) ' multiset combinations -> 220
=MULTINOMIAL(3,3,1,2,1) ' 10!/(3!3!1!2!1!) -> 50400
' ── Applied ─────────────────────────────────────────────────────────
=COMBIN(8,3)*COMBIN(6,2) ' committees with exactly 3 women -> 840
=COMBIN(8,3)*COMBIN(6,2)/COMBIN(14,5) ' probability -> 0.4196
=1/COMBIN(49,6) ' lottery jackpot -> 7.151E-08
=4*COMBIN(13,5)/COMBIN(52,5) ' P(flush) -> 0.00198
R¶
# ── Bayes' theorem ─────────────────────────────────────────────────
bayes <- function(prior, sens, fpr) {
evidence <- sens * prior + fpr * (1 - prior)
list(evidence = evidence, posterior = sens * prior / evidence)
}
bayes(prior = 0.01, sens = 0.95, fpr = 0.10)
# $evidence 0.1085
# $posterior 0.08756
# Natural-frequency table — the intuition check
N <- 10000; p <- 0.01; sens <- 0.95; spec <- 0.90
tab <- matrix(c(N*p*sens, N*(1-p)*(1-spec),
N*p*(1-sens), N*(1-p)*spec),
nrow = 2, byrow = TRUE,
dimnames = list(test = c("Positive", "Negative"),
truth = c("Disease", "No disease")))
addmargins(tab)
tab["Positive","Disease"] / sum(tab["Positive", ]) # 0.08756
# General k-category version
priors <- c(A = 0.60, B = 0.40)
likelihoods <- c(A = 0.02, B = 0.05) # P(defective | machine)
post <- priors * likelihoods / sum(priors * likelihoods)
round(post, 4) # A 0.375 B 0.625
# ── Counting rules ─────────────────────────────────────────────────
factorial(5) # 120
choose(10, 3) # 120 combinations (nCr)
factorial(10) / factorial(7) # 720 permutations (nPr)
perm <- function(n, r) factorial(n) / factorial(n - r)
perm(10, 3) # 720
# Arrangements with repeats: STATISTICS
factorial(10) / prod(factorial(c(3, 3, 1, 2, 1))) # 50400
# ── Applied ────────────────────────────────────────────────────────
choose(8, 3) * choose(6, 2) # 840
choose(8, 3) * choose(6, 2) / choose(14, 5) # 0.4196
1 / choose(49, 6) # 7.151e-08
4 * choose(13, 5) / choose(52, 5) # 0.00198
# Enumerate rather than count, when n is small
combn(5, 2) # all 10 pairs from {1..5}
ncol(combn(5, 2)) # 10
Python¶
import numpy as np
from math import factorial, comb, perm
from itertools import combinations, permutations
# ── Bayes' theorem ─────────────────────────────────────────────────
def bayes(prior, sens, fpr):
evidence = sens * prior + fpr * (1 - prior)
return evidence, sens * prior / evidence
bayes(0.01, 0.95, 0.10) # (0.1085, 0.08756...)
# Natural-frequency table
N, p, sens, spec = 10_000, 0.01, 0.95, 0.90
tp = N * p * sens # 95
fp = N * (1 - p) * (1 - spec) # 990
tp / (tp + fp) # 0.08756
# General k-category version
priors = np.array([0.60, 0.40]) # machine A, B
likelihoods = np.array([0.02, 0.05]) # P(defective | machine)
post = priors * likelihoods / (priors * likelihoods).sum()
post.round(4) # [0.375, 0.625]
# ── Counting rules ─────────────────────────────────────────────────
factorial(5) # 120
comb(10, 3) # 120 combinations (nCr)
perm(10, 3) # 720 permutations (nPr)
# Arrangements with repeats: STATISTICS
factorial(10) // (factorial(3) * factorial(3) * factorial(2)) # 50400
# ── Applied ────────────────────────────────────────────────────────
comb(8, 3) * comb(6, 2) # 840
comb(8, 3) * comb(6, 2) / comb(14, 5) # 0.4196
1 / comb(49, 6) # 7.151e-08
4 * comb(13, 5) / comb(52, 5) # 0.00198
# Enumerate rather than count, when n is small
list(combinations(range(5), 2)) # 10 pairs
len(list(permutations(range(5), 2))) # 20 ordered pairs
Quick Reference¶
| Task | Excel | R | Python |
|---|---|---|---|
| Factorial | FACT(n) |
factorial(n) |
math.factorial(n) |
Permutations nPr |
PERMUT(n, r) |
factorial(n)/factorial(n-r) |
math.perm(n, r) |
Combinations nCr |
COMBIN(n, r) |
choose(n, r) |
math.comb(n, r) |
| With repetition | PERMUTATIONA / COMBINA |
n^r / choose(n+r-1, r) |
n**r / comb(n+r-1, r) |
| Multiset arrangements | MULTINOMIAL(k1,k2,…) |
factorial(n)/prod(factorial(k)) |
manual |
| Enumerate combinations | — | combn(n, r) |
itertools.combinations |
| Bayes posterior | sens*prior/(sens*prior+fpr*(1-prior)) |
user function | user function |
Common Mistakes¶
- Base-rate neglect — quoting the test's accuracy as the probability of disease. Always fold in the prior.
- Using a permutation where order does not matter (over-counts by a factor of
r!). - Forgetting
0! = 1. - Adding instead of multiplying across independent stages of a counting problem.
- In "exactly k of type X" problems, forgetting to multiply by the ways of choosing the other type.
- Treating
P(A|B)andP(B|A)as interchangeable.
Exercises: 05-03: Exercises — Bayes' Theorem and Counting Rules
⬅️ Previous: 05-02: Probability Rules and Conditional Probability ➡️ Next: 06-01: Random Variables and Expected Value