A display engine needs each memory read back within 2 µs, or its line buffer runs dry and the screen glitches. The memory system's average read latency is 400 ns, five times better than needed. The screen still glitches, a few times a minute. The average was never the problem. One read in a few thousand takes longer than 2 µs, and one is enough.

Why is the average the wrong number for latency, and what should be specified instead?

The obvious answer, and why it breaks

The obvious summary of latency is its average. It is easy to measure and easy to compare. But a requester with a deadline does not care about the average: it fails on the slowest request it sees. And when a single operation needs many requests (a cache-line fill, a packet assembled from several reads, a page that waits for many servers), it is as slow as the slowest of them. The rare slow requests, the tail, decide whether the system meets its goals.

The idea: percentiles

• The p50 (median) is the latency half of requests beat.

• The p99 is the latency 99% of requests beat: 1 request in 100 is slower.

• The p99.9 and beyond describe rarer and slower requests: 1 in 1,000, 1 in 10,000.

A specification such as "p99 read latency ≤ 1 µs at 70% load" says what fraction of requests may be slow, by how much, and under what load, which an average cannot.

Build it: average against tail

The model is the single queue from lesson 6.11, at four loads. It records every request's latency and reports the mean, the median and the 99th percentile, in units of the average service time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

"""Average and tail latency of a single queue (random arrivals, random service).

Same queue as lesson 6.11: exponential gaps at rate rho, exponential service with
mean 1. The model records every request's latency and reports the median (p50) and
the 99th percentile (p99): the latency that 1 request in 100 exceeds.
"""
import random


def latencies(rho, n=200000):
    random.seed(5)
    t = free_at = 0.0
    out = []
    for _ in range(n):
        t += random.expovariate(rho)
        start = max(t, free_at)
        free_at = start + random.expovariate(1.0)
        out.append(free_at - t)
    return sorted(out)


def pct(xs, p):
    return xs[int(p / 100 * (len(xs) - 1))]


loads = [0.5, 0.7, 0.8, 0.9]
print("load    mean    p50    p99   p99/mean")
rows = []
for r in loads:
    xs = latencies(r)
    mean = sum(xs) / len(xs)
    rows.append((r, mean, pct(xs, 50), pct(xs, 99)))
    print(f"{r:4.1f} {mean:7.2f} {pct(xs, 50):6.2f} {pct(xs, 99):6.2f} {pct(xs, 99) / mean:8.1f}")
assert all(p99 > 4 * mean for _, mean, _, p99 in rows)
print("PASS: the 99th percentile is several times the mean at every load, and grows with it")

It prints:

load    mean    p50    p99   p99/mean
 0.5    1.99   1.37   9.27      4.7
 0.7    3.29   2.26  15.32      4.7
 0.8    4.89   3.37  22.76      4.7
 0.9    9.50   6.49  42.35      4.5
PASS: the 99th percentile is several times the mean at every load, and grows with it

The median is below the mean, because a few very slow requests pull the mean up. The p99 is between 4.5 and 4.7 times the mean at every load here. In this queue, a design that meets its latency target "on average" with less than about 4.5 times margin misses it for 1 request in 100.

What makes the tail

Fan-out deserves a number. If each request has a 1% chance of being slow, an operation that needs 100 of them meets at least one slow request with probability 1 − 0.99¹⁰⁰ ≈ 63%. The p99 of the part becomes the median of the whole.

How architects use it

• Specify percentiles under load, not averages at idle, for every latency-critical path.

• Give urgent traffic its own resources or priority, so its tail does not include other traffic's bursts (Part 8).

• Bound the rare events: maximum refresh blocking, maximum retry time, maximum arbitration wait.

• Measure histograms, not just means, in simulation and on silicon.

What breaks

The opening display (illustrative): the memory controller occasionally held a read behind a long batch of writes it was draining for efficiency (lesson 6.10). The average was fine; the worst-case wait behind a full write batch exceeded 2 µs. The fix capped the write batch length and gave display reads priority over write draining, which raised the average latency slightly and cut the tail below 1 µs.

Key takeaways

• Requesters with deadlines, and operations that fan out, fail on the slowest requests, not the average.

• Specify percentiles (p99, p99.9) at a stated load, and measure histograms.

• Bound the causes of the tail: bursts, rare slow events, head-of-line blocking, unfair arbitration.

Related topics

• Part 6 overview: Concurrency and Performance