04-02: Measures of Position and Outliers¶
A centre and a spread describe the whole data set. A measure of position describes where one particular value sits inside it — and that is what lets you compare a score on one test with a score on a completely different one.
Z-Scores (Standard Scores)¶
The number of standard deviations a value lies from the mean.
| z | Meaning |
|---|---|
z = 0 |
The value is the mean |
z = +1.5 |
1.5 standard deviations above the mean |
z = −2.0 |
2 standard deviations below the mean |
\|z\| > 2 |
Unusual |
\|z\| > 3 |
Very unusual — a common outlier flag |
Z-scores are unitless, which is the whole point: they let you compare values measured on different scales.
Example. Which is the better performance?
Exam A: score 82, μ = 75, σ = 5 → z = (82 − 75) / 5 = +1.40
Exam B: score 88, μ = 80, σ = 10 → z = (88 − 80) / 10 = +0.80
The 82 is the stronger result — it is further above its own mean in standard-deviation terms, even though 88 is the bigger raw number.
Standardizing a whole variable (subtracting the mean and dividing by the SD) leaves a data set with mean 0 and standard deviation 1. This is the exact operation behind the normal-table work in 07-02.
Percentiles¶
The k-th percentile Pk is the value below which about k percent of the data lies.
Finding the percentile rank of a value:
Finding the value at a given percentile (by hand):
Step 1 Sort the data ascending.
Step 2 Compute the position c = (n · k) / 100
Step 3 If c is NOT a whole number → round UP; that position is Pk.
If c IS a whole number → Pk = average of positions c and c+1.
Example. Data: 2, 3, 5, 6, 8, 10, 12, 15, 18, 20 (n = 10). Find P25.
Find P50:
c = (10 × 50) / 100 = 5 → whole → average positions 5 and 6 = (8 + 10)/2 = 9
P50 = 9 (this is also the median)
Quartiles and the Five-Number Summary¶
Quartiles split the sorted data into four equal parts.
Q1 = P25 the 25th percentile (lower quartile)
Q2 = P50 the median
Q3 = P75 the 75th percentile (upper quartile)
Interquartile range IQR = Q3 − Q1 the middle 50% of the data
Semi-interquartile range = IQR / 2
Midquartile = (Q1 + Q3) / 2
The five-number summary — min, Q1, median, Q3, max — is what a boxplot draws (02-02).
Note
Quartiles are not uniquely defined. There are at least nine accepted methods, and different software picks different defaults. Excel's QUARTILE.INC and R's quantile(type = 7) agree; Excel's QUARTILE.EXC matches R's type = 6. Small data sets can give visibly different Q1/Q3 across tools — always say which method you used.
Deciles and Other Fractiles¶
| Fractile | Splits into | Notation |
|---|---|---|
| Quartiles | 4 parts | Q1, Q2, Q3 |
| Deciles | 10 parts | D1 … D9 (D5 = median) |
| Percentiles | 100 parts | P1 … P99 |
Deciles are common in income and test-score reporting: "the top decile" is the highest 10%.
Identifying Outliers¶
An outlier is a value far from the rest of the data. Two standard rules:
1. The 1.5 × IQR rule (used by boxplots — resistant, preferred)¶
Lower fence = Q1 − 1.5 × IQR
Upper fence = Q3 + 1.5 × IQR
Any value outside the fences is an outlier.
Beyond 3 × IQR it is often called an EXTREME outlier.
2. The z-score rule (assumes roughly bell-shaped data)¶
The z-score rule is less reliable on small samples, because a single extreme value inflates s and thereby hides itself. The IQR rule is built from resistant quantities and does not suffer from this.
What to do with an outlier¶
- Check for an error — data entry, wrong units, a code like 999 for "missing". Fix or remove it.
- Check whether it is a different population — a wholesale order in retail data.
- If it is genuine, keep it and report both analyses (with and without), or use resistant statistics (median, IQR).
Warning
Never delete an outlier just because it is inconvenient. That is data manipulation, not cleaning.
Worked Example¶
Twelve monthly electricity bills (dollars):
Quartiles (Excel QUARTILE.INC / R type = 7):
Fences:
Lower fence = 73.25 − 1.5 × 23.25 = 73.25 − 34.875 = 38.38
Upper fence = 96.50 + 1.5 × 23.25 = 96.50 + 34.875 = 131.38
187 > 131.38 → 187 is an OUTLIER
Z-score check on 187:
The z-rule (|z| > 3) just fails to flag it — precisely because that one value inflated s. This is the masking effect the IQR rule avoids.
Five-number summary: 62, 73.25, 84.5, 96.5, 187
Percentile rank of 90:
Excel¶
' ── Z-scores ────────────────────────────────────────────────────────
=(A2-AVERAGE($A$2:$A$13))/STDEV.S($A$2:$A$13) ' z for one value
=STANDARDIZE(A2, AVERAGE($A$2:$A$13), STDEV.S($A$2:$A$13)) ' same thing
=ABS(B2)>3 ' outlier flag
' ── Percentiles ─────────────────────────────────────────────────────
=PERCENTILE.INC(A2:A13, 0.25) ' inclusive method (matches R type 7)
=PERCENTILE.EXC(A2:A13, 0.25) ' exclusive method (matches R type 6)
=PERCENTRANK.INC(A2:A13, 90) ' percentile rank of the value 90
=PERCENTRANK.EXC(A2:A13, 90)
' ── Quartiles and IQR ───────────────────────────────────────────────
=QUARTILE.INC(A2:A13, 1) ' Q1 -> 73.25
=QUARTILE.INC(A2:A13, 2) ' median -> 84.5 (= MEDIAN)
=QUARTILE.INC(A2:A13, 3) ' Q3 -> 96.5
=QUARTILE.INC(A2:A13,3)-QUARTILE.INC(A2:A13,1) ' IQR -> 23.25
' ── Five-number summary ─────────────────────────────────────────────
=MIN(A2:A13)
=QUARTILE.INC(A2:A13,1)
=MEDIAN(A2:A13)
=QUARTILE.INC(A2:A13,3)
=MAX(A2:A13)
' ── Outlier fences and flag ─────────────────────────────────────────
=QUARTILE.INC($A$2:$A$13,1)-1.5*(QUARTILE.INC($A$2:$A$13,3)-QUARTILE.INC($A$2:$A$13,1))
=QUARTILE.INC($A$2:$A$13,3)+1.5*(QUARTILE.INC($A$2:$A$13,3)-QUARTILE.INC($A$2:$A$13,1))
=IF(OR(A2<$E$1, A2>$E$2), "OUTLIER", "") ' E1 = lower fence, E2 = upper
' ── Rank and deciles ────────────────────────────────────────────────
=RANK.EQ(A2, $A$2:$A$13, 1) ' 1 = smallest; use 0 for largest first
=LARGE(A2:A13, 3) ' 3rd largest
=SMALL(A2:A13, 3) ' 3rd smallest
=PERCENTILE.INC(A2:A13, 0.9) ' D9 — the top decile boundary
R¶
bills <- c(62, 68, 71, 74, 79, 83, 86, 90, 95, 101, 108, 187)
# ── Z-scores ───────────────────────────────────────────────────────
z <- scale(bills) # returns a matrix; mean 0, sd 1
z <- as.numeric(scale(bills))
round(z, 2)
bills[abs(z) > 3] # z-rule outliers (none here)
# ── Percentiles and quartiles ──────────────────────────────────────
quantile(bills) # 0, 25, 50, 75, 100 (type 7)
quantile(bills, c(0.10, 0.25, 0.90))
quantile(bills, 0.25, type = 6) # matches Excel's QUARTILE.EXC
median(bills)
IQR(bills) # 23.25
# Percentile rank of a value
mean(bills < 90) * 100 # proportion strictly below -> 58.3
(sum(bills < 90) + 0.5) / length(bills) * 100 # textbook formula -> 62.5
# ── Five-number summary ────────────────────────────────────────────
fivenum(bills) # Tukey's hinges (used by boxplot whiskers)
summary(bills) # min, Q1, median, mean, Q3, max
# ── Outlier detection: 1.5 × IQR rule ──────────────────────────────
Q1 <- quantile(bills, 0.25); Q3 <- quantile(bills, 0.75)
iqr <- Q3 - Q1
lower <- Q1 - 1.5 * iqr; upper <- Q3 + 1.5 * iqr
c(lower = lower, upper = upper) # 38.375, 131.375
bills[bills < lower | bills > upper] # 187
# boxplot.stats does it for you
boxplot.stats(bills)$out # 187
# ── Ranks and extremes ─────────────────────────────────────────────
rank(bills)
sort(bills, decreasing = TRUE)[3] # 3rd largest
Python¶
import numpy as np
import pandas as pd
from scipy import stats
bills = pd.Series([62, 68, 71, 74, 79, 83, 86, 90, 95, 101, 108, 187])
# ── Z-scores ───────────────────────────────────────────────────────
z = (bills - bills.mean()) / bills.std() # pandas std -> sample (ddof=1)
z = pd.Series(stats.zscore(bills, ddof=1)) # same
bills[z.abs() > 3] # z-rule outliers (none here)
# ── Percentiles and quartiles ──────────────────────────────────────
bills.quantile([0, 0.25, 0.5, 0.75, 1]) # linear method = Excel .INC
np.percentile(bills, [10, 25, 90])
bills.quantile(0.25, interpolation="linear")
Q1, Q3 = bills.quantile(0.25), bills.quantile(0.75)
iqr = Q3 - Q1 # 23.25
stats.iqr(bills) # same
# Percentile rank of a value
stats.percentileofscore(bills, 90, kind="mean") # 62.5
stats.percentileofscore(bills, 90, kind="strict") # 58.3
# ── Five-number summary ────────────────────────────────────────────
bills.describe()
np.percentile(bills, [0, 25, 50, 75, 100])
# ── Outlier detection: 1.5 × IQR rule ──────────────────────────────
lower, upper = Q1 - 1.5 * iqr, Q3 + 1.5 * iqr # 38.375, 131.375
outliers = bills[(bills < lower) | (bills > upper)]
print(outliers.tolist()) # [187]
# Flag them in a DataFrame
df = bills.to_frame("bill")
df["is_outlier"] = (df["bill"] < lower) | (df["bill"] > upper)
# ── Ranks and extremes ─────────────────────────────────────────────
bills.rank()
bills.nlargest(3)
bills.nsmallest(3)
Quick Reference¶
| Task | Excel | R | Python |
|---|---|---|---|
| Z-score | STANDARDIZE(x, mean, sd) |
scale(x) |
stats.zscore(x, ddof=1) |
| Percentile value | PERCENTILE.INC(range, k) |
quantile(x, k) |
x.quantile(k) |
| Percentile rank | PERCENTRANK.INC(range, x) |
mean(x < v)*100 |
stats.percentileofscore |
| Q1 / Q3 | QUARTILE.INC(range, 1/3) |
quantile(x, c(.25,.75)) |
x.quantile([.25,.75]) |
| IQR | QUARTILE(.,3)-QUARTILE(.,1) |
IQR(x) |
stats.iqr(x) |
| Five-number summary | MIN/QUARTILE/MEDIAN/MAX | fivenum(x) / summary(x) |
x.describe() |
| Outlier fences | Q1-1.5*IQR, Q3+1.5*IQR |
manual or boxplot.stats |
manual |
| Boxplot outliers | Insert ▸ Box and Whisker | boxplot.stats(x)$out |
ax.boxplot(x) |
| Rank | RANK.EQ |
rank(x) |
x.rank() |
| k-th largest | LARGE(range, k) |
sort(x, TRUE)[k] |
x.nlargest(k) |
Common Mistakes¶
- Comparing raw scores from two different tests instead of comparing z-scores.
- Using the population SD in a z-score when the data is a sample (or vice versa) without saying which.
- Assuming Q1 from Excel
QUARTILE.EXCwill match R's default — it will not;QUARTILE.INCdoes. - Using the
|z| > 3rule on a small sample, where the outlier itself inflatessand hides. - Deleting outliers without investigating them first.
- Reporting "the 90th percentile is 90% of the maximum" — percentiles are positions, not proportions of the maximum.
Exercises: 04-02: Exercises — Measures of Position and Outliers
⬅️ Previous: 04-01: Measures of Variation ➡️ Next: 05-01: Probability Basics