Two teams argue about the unit size of a new on-chip network. One wants small units so that short control messages waste little space. The other wants large units so that the per-unit header and CRC are paid less often. Both show spreadsheets proving their case. They are both right, for different traffic.
How should messages be cut into fixed-size units on a link, and how big should those units be?
The obvious answer is to send variable-length packets exactly as long as each message, so nothing is wasted. That is fine for framing (Part 2's other lessons) but awkward for the machinery underneath: buffers, credits and error correction all prefer fixed-size units. A credit that means "one slot of unknown size" forces buffers sized for the largest packet; FEC needs fixed-size blocks to work on. So links cut traffic into fixed-size units anyway, and the question becomes how big.
Unit | What it is | Why it exists |
|---|---|---|
Packet (or message) | What the upper layer wants to send, any length | Meaning: one request, one response |
Flit (flow-control unit) | A fixed-size piece of a packet; the unit buffers and credits count | Simple buffers and credits; a natural block for CRC and FEC |
Phit (physical unit) | What the wires carry in one cycle | Width of the physical channel; a flit may take several phits |
Every flit carries some overhead (a small header, a CRC), and a packet that does not fill its last flit wastes the rest. So the efficiency depends on both the flit size and the mix of message sizes, which is exactly what the two teams were arguing about.
The model computes efficiency (message bytes divided by bytes on the wire) for three flit sizes and seven message sizes, first with one message per flit, then with many small messages packed into shared flits. The 256-byte flit uses 20 bytes of overhead, matching a PCIe 6.0 flit's 236 bytes for transaction-layer data out of 256.
"""Packing messages into fixed-size flits.
Each flit has F bytes, of which H are overhead (header, CRC). A message that does not
fill its last flit wastes the rest. Efficiency = message bytes / bytes on the wire.
Flit sizes: 16 B (2 B overhead), 64 B (4 B), and 256 B with 20 B of overhead, which
matches a PCIe 6.0 flit's 236 bytes for TLPs out of 256.
"""
import math
FLITS = [(16, 2), (64, 4), (256, 20)]
SIZES = [8, 16, 32, 64, 128, 256, 512]
def efficiency(msg, F, H, packed):
if packed: # many messages share flits: only per-flit overhead counts
return (F - H) / F
return msg / (math.ceil(msg / (F - H)) * F)
print("message " + " ".join(f"{F:>4} B flit" for F, _ in FLITS))
table = {}
for m in SIZES:
row = [efficiency(m, F, H, packed=False) for F, H in FLITS]
table[m] = row
print(f"{m:5d} B " + " ".join(f"{100 * e:9.1f}%" for e in row))
print("packed " + " ".join(f"{100 * efficiency(0, F, H, True):9.1f}%" for F, H in FLITS))
packed = [efficiency(0, F, H, True) for F, H in FLITS]
assert all(table[m][0] >= table[m][2] for m in SIZES) and packed[2] > packed[0]
print("PASS: with one message per flit the 256 B flit is never best; packed, it beats the 16 B flit")It prints:
message 16 B flit 64 B flit 256 B flit
8 B 50.0% 12.5% 3.1%
16 B 50.0% 25.0% 6.2%
32 B 66.7% 50.0% 12.5%
64 B 80.0% 50.0% 25.0%
128 B 80.0% 66.7% 50.0%
256 B 84.2% 80.0% 50.0%
512 B 86.5% 88.9% 66.7%
packed 87.5% 93.8% 92.2%
PASS: with one message per flit the 256 B flit is never best; packed, it beats the 16 B flitTwo conclusions. With one message per flit, the largest flit is the least efficient at every size in this table: it wastes most of its space on small messages, and even a 512-byte message leaves a partly empty third 256-byte flit. The 16-byte flit leads up to 256 bytes, and the 64-byte flit only overtakes it at 512 bytes. Large flits pay off when many messages share flits, the packed row, where their smaller per-byte overhead wins (92% for 256 B against 87.5% for 16 B). So a large-flit design is only efficient if it packs small messages together, which adds logic and some latency.
Flit size | Wins when | Costs |
|---|---|---|
Small (a few bytes) | Traffic is many small messages; latency matters (a flit is ready sooner) | Per-flit overhead paid often; more flits to track |
Large (hundreds of bytes) | Traffic is packed or mostly large; strong per-flit FEC and CRC are needed | Packing logic; latency waiting to fill a flit; waste when traffic is sparse |
Phit = flit | Simple: one flit per cycle | Wide wires, or small flits |
Flit = several phits | Narrow physical channels | Serialisation latency per flit |
Protocol | Its choice |
|---|---|
PCI Express Gen1–5 | Variable-length packets on the wire, with credits counting header and 16-byte data units (lesson 4.6). |
PCI Express Gen6 | Fixed 256-byte flits: 236 bytes of transaction-layer data, 6 bytes of data-link payload, 8 bytes of CRC and 6 bytes of FEC. Fixed-size flits let the FEC that PAM4 signalling requires work on fixed blocks; errors are corrected inside the flit, detected by the CRC and recovered by flit-level retry. |
On-chip networks | Flits sized to the router datapath, with packets split into a head flit, body flits and a tail flit; buffers and credits count flits (Parts 4 and 8). |
A design adopts large fixed flits for their low overhead and sends each small control message in its own flit (illustrative). Control traffic, mostly 8 to 16-byte messages, now uses 256 bytes of link bandwidth each, about 3 to 6% efficient by the model's numbers, and under heavy control traffic the link saturates at a fraction of its rated payload. The fix is packing several messages into each flit, which is what large-flit protocols are designed around.
Each flit carries 60 bytes of message, so 100 bytes need 2 flits = 128 bytes on the wire: 100/128 ≈ 78%.
Four phits per flit, which adds serialisation latency: the receiver has the whole flit only after four cycles.
Their fixed size and overhead are amortised only when they are full; one small message per large flit wastes most of it.
A packet carries meaning, a flit is the fixed unit buffers, credits and FEC work on, and a phit is what the wires carry per cycle.
Small flits waste little on small messages; large flits have lower overhead per byte but are efficient only when messages are packed into them.
PCIe 6.0 uses fixed 256-byte flits because PAM4 needs FEC, and FEC needs fixed-size blocks.