An interconnect link runs at 50% utilisation and its average latency is 20 ns. Traffic grows to 90% and latency becomes 100 ns. Grow it to 95% and latency doubles again. Nothing changed in the link; the extra load was less than doubled. Latency did not rise in proportion to load. It rose like a wall.

Why does latency explode near full load, and how close to full can a resource be run?

The obvious answer, and why it breaks

The obvious expectation is that a resource with spare capacity adds no delay: at 90% busy, there is still 10% free, so requests should get through. That would be true if requests arrived evenly spaced. They do not. They arrive in clumps, and a clump that arrives while the resource is busy waits. The busier the resource, the more likely each request finds a queue in front of it, and the longer that queue.

The idea: waiting grows as 1 / (1 − load)

For a single server with random arrivals and random service times, the average time in the system (waiting plus service), measured in service times, is:

Other arrival and service patterns change the constant but not the shape: smoother traffic and more regular service lower the curve, burstier traffic raises it, and every curve turns upwards sharply as ρ approaches 1. The knee is where waiting starts to dominate service, usually somewhere between 70% and 85% utilisation.

Build it: a single-queue simulator

The model sends 200,000 requests with random (exponential) gaps through one server with random (exponential) service times of mean 1, at several loads, and compares the measured average latency with the formula.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

"""Latency against load for a single queue (random arrivals, random service).

Requests arrive at random (exponential gaps) at rate rho per unit time and are
served one at a time, each taking a random time with mean 1 (exponential). The
model measures the average time a request spends waiting plus being served, and
compares it with the formula for this queue, 1 / (1 - rho).
"""
import random


def mean_latency(rho, n=200000):
    random.seed(3)
    t = free_at = total = 0.0
    for _ in range(n):
        t += random.expovariate(rho)             # next arrival
        start = max(t, free_at)
        free_at = start + random.expovariate(1.0)
        total += free_at - t
    return total / n


loads = [0.1, 0.3, 0.5, 0.7, 0.8, 0.9, 0.95]
print("load:     " + " ".join(f"{r:6.2f}" for r in loads))
meas = [mean_latency(r) for r in loads]
print("measured: " + " ".join(f"{m:6.2f}" for m in meas))
print("formula:  " + " ".join(f"{1 / (1 - r):6.2f}" for r in loads))
for r, m in zip(loads, meas):
    assert abs(m - 1 / (1 - r)) / (1 / (1 - r)) < 0.12
print("PASS: latency is about 1/(1-load): double at 50%, 10x at 90%, 20x at 95%")

It prints:

load:       0.10   0.30   0.50   0.70   0.80   0.90   0.95
measured:   1.11   1.43   2.00   3.35   5.06  10.23  19.82
formula:    1.11   1.43   2.00   3.33   5.00  10.00  20.00
PASS: latency is about 1/(1-load): double at 50%, 10x at 90%, 20x at 95%

At 50% load a request already spends as long waiting as being served. From 80% to 95% load, a 19% increase in traffic quadruples the latency. That is the wall in the opening story.

What this means for design

What breaks

A memory subsystem (illustrative) was sized so the average bandwidth demand was 92% of sustained capacity, "leaving 8% headroom". Average latency was several times the idle figure, and every burst of traffic pushed it higher. A latency-sensitive display engine sharing the memory underflowed its buffer during bursts. Adding a second channel brought utilisation to about 46% and latency close to idle.

Key takeaways

• Random arrivals make queues form well before full load; latency grows as 1 / (1 − load) for a simple queue.

• The knee, typically 70–85% utilisation, is where waiting starts to dominate.

• Plan shared resources below the knee and quote latency at load.

Related topics

• Part 6 overview: Concurrency and Performance