A PCI Express Gen3 x4 link is "32 gigatransfers per second", or 4 GB/s. A storage device behind it measures 3.3 GB/s of data with small transfers and 3.7 GB/s with large ones. Nothing is broken. Between the line rate and the data, every layer of the protocol takes its share, and some shares are paid per packet.

How much of a link's line rate becomes useful data, and where does the rest go?

The obvious answer, and why it breaks

The obvious estimate of useful bandwidth is the line rate, perhaps less the encoding overhead quoted on the datasheet. But encoding is only the first layer. Each packet also carries framing, a header, a check code, and shares the link with acknowledgements and credit updates. Some of these costs are a fixed fraction (encoding), and some are fixed per packet (headers), which makes small packets far less efficient than large ones.

The idea: the overhead stack

Because the per-packet costs are fixed, efficiency for payload P and per-packet overhead O is P / (P + O): halving the payload size can cost several percent.

Build it: an efficiency calculator for PCIe Gen3 x4

The model starts from the line rate and removes each layer: 128b/130b encoding, 24 bytes of overhead per packet (a 4-byte start token carrying the sequence number, a 16-byte header, a 4-byte CRC), and link-management packets, assumed to be one 8-byte packet per four data packets (illustrative).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

"""From line rate to payload bandwidth on a PCIe Gen3 x4 link.

Gen3 runs 8 GT/s per lane with 128b/130b encoding. Each TLP carries a 4-byte STP
token (with the sequence number), a 4-DW (16-byte) header and a 4-byte LCRC: 24
bytes of overhead (no ECRC). Data link layer packets (acknowledgements and credit
updates) also use the link; here one 8-byte DLLP for every 4 TLPs (illustrative).
"""
LANES, GT = 4, 8e9
line = LANES * GT / 8                         # bytes per second on the wire
coded = line * 128 / 130                      # after 128b/130b encoding
TLP_OVERHEAD, DLLP, TLPS_PER_DLLP = 24, 8, 4


def payload_rate(payload):
    per_tlp = payload + TLP_OVERHEAD + DLLP / TLPS_PER_DLLP
    return coded * payload / per_tlp


print(f"line rate          {line / 1e9:.3f} GB/s")
print(f"after 128b/130b    {coded / 1e9:.3f} GB/s")
for mps in (128, 256, 512):
    after_tlp = coded * mps / (mps + TLP_OVERHEAD)
    final = payload_rate(mps)
    print(f"payload {mps:>3} B: after TLP overhead {after_tlp / 1e9:.3f} GB/s, after DLLPs {final / 1e9:.3f} GB/s ({final / line * 100:.1f}% of line rate)")
    assert final < after_tlp < coded < line
assert payload_rate(512) > payload_rate(256) > payload_rate(128)
print("PASS: bigger payloads amortise the fixed per-packet overhead; encoding costs 1.5% at every size")

It prints:

line rate          4.000 GB/s
after 128b/130b    3.938 GB/s
payload 128 B: after TLP overhead 3.317 GB/s, after DLLPs 3.274 GB/s (81.8% of line rate)
payload 256 B: after TLP overhead 3.601 GB/s, after DLLPs 3.575 GB/s (89.4% of line rate)
payload 512 B: after TLP overhead 3.762 GB/s, after DLLPs 3.748 GB/s (93.7% of line rate)
PASS: bigger payloads amortise the fixed per-packet overhead; encoding costs 1.5% at every size

Encoding takes 1.5% regardless. The per-packet overhead takes 15.8% at 128 bytes but only 4.5% at 512 bytes. The link's maximum payload size, negotiated between its two ends, is therefore one of the biggest levers on throughput, and real traffic with many small packets sits nearer the 128-byte line.

Other places efficiency goes

• Reads cost twice: a read request is a packet with a header and no data, travelling the other way, before the completion carries the data back.

• Credits and outstanding limits: if the receiver's credits or the requester's tags do not cover the round trip, the link idles between packets (lessons 4.5 and 6.4).

• Bus turnaround: shared, bidirectional buses lose cycles every time the direction changes (lesson 6.10).

How real links compare

What breaks

A storage controller (illustrative) was benchmarked with 4 KB transfers and promised 3.7 GB/s. In the product, a database issued mostly 128-byte reads, each a separate request and completion, and the controller delivered well under 3 GB/s. The benchmark measured the protocol at its most efficient payload size; the workload ran it at its least. Quote throughput with the payload size it was measured at.

Key takeaways

• Useful bandwidth is line rate minus encoding, per-packet framing, headers and CRCs, and link-management traffic.

• Per-packet overhead makes small payloads expensive: efficiency is P / (P + O).

• Quote throughput with the payload size and traffic pattern it was measured at.

Related topics

• Part 6 overview: Concurrency and Performance