A team upgrades a link from 8 GB/s to 16 GB/s. Their processor's cache misses, which fetch one 64-byte line at a time, get no faster at all. Another team, streaming video frames over the same upgrade, gets twice the frame rate. Both measured "performance". They measured different things.
What exactly are latency, bandwidth and throughput, and why does improving one not always help?
The obvious view is that a faster link is faster for everything. But "faster" hides three different questions: how long does one item take to arrive, how much can the link carry per second, and how much does it actually carry? A cache miss waits for one line to arrive, so it cares about the first question. A video stream cares about the second. Doubling the link's width answers the second and barely touches the first.
Quantity | Measures | Units | Like a pipe's |
|---|---|---|---|
Latency | Time for one item to get from request to arrival | Nanoseconds or cycles | Length: how long water takes to come out after you open the tap |
Bandwidth | The most the channel can carry per unit time | Bytes per second, or bits per second | Width: how much water can flow |
Throughput | What is actually carried per unit time | Bytes per second, or items per second | The flow you actually get, often less than the width allows |
A link with illustrative numbers: 16 GB/s bandwidth and 500 ns latency, fetching 64-byte lines.
Time to push one 64-byte line onto the link: 64 B / 16 GB/s = 4 ns
Time for that line to arrive: 500 ns latency + 4 ns = 504 ns
One line at a time: 64 B per 504 ns = 0.127 GB/s (0.8% of the bandwidth)
Double the bandwidth to 32 GB/s: 500 + 2 ns = 502 ns per line: 0.4% faster
To use the full 16 GB/s, keep 504 ns x 16 GB/s = 8,064 bytes in flight = 126 linesThe single-line requester is latency-bound: almost all of its time is spent waiting, and bandwidth hardly matters. A streaming requester that keeps many lines in flight is bandwidth-bound. Throughput connects the two: it depends on the bandwidth, the latency, and how much is in flight at once (lesson 6.2).
Pipelining a link raises its clock frequency, and so its bandwidth, but each stage adds a cycle of latency.
Bigger packets spread header overhead and raise effective bandwidth (lesson 6.9), but a small urgent message waits behind a big one.
Batching (credit returns, interrupts, doorbells) saves bandwidth and adds latency.
Running a link fully loaded maximises throughput and makes queues, and latency, grow (lesson 6.11).
Quantity | Measure by |
|---|---|
Latency | Timestamp a single request and its response with nothing else in flight; then repeat under load, because latency rises with load |
Bandwidth | From the specification: width × clock rate, less encoding; it is a ceiling, not a measurement |
Throughput | Count bytes delivered over a long interval of real traffic |
The opening upgrade (illustrative): the processor's misses were latency-bound, with one or two lines outstanding at a time, so doubling the bandwidth did nothing for them. What helped was letting each core keep more misses outstanding and adding a prefetcher, which turned a latency-bound pattern into a bandwidth-bound one.
32 GB/s × 250 ns = 8,000 bytes (125 lines of 64 bytes).
About 64 bytes per microsecond, 64 MB/s, because it is latency-bound.
Latency is time per item; bandwidth is the channel's capacity; throughput is what is actually delivered.
A requester with little in flight is latency-bound and gains nothing from more bandwidth.
Pipelining, packet size, batching and load all trade one quantity against another.