A link doubles its speed by switching from two signal levels to four. The price is a raw bit error rate thousands of times worse. With replay alone, packets would now be resent constantly, and each replay costs a round trip of bandwidth and latency. The link needs to fix most errors where they land, without asking the sender for anything.
When should a link correct errors itself instead of asking for a resend?
The obvious recovery is detection plus replay: a CRC catches the bad packet and the sender resends it (lesson 10.11). That is cheap when errors are rare: almost no bandwidth is spent on redundancy, and the occasional replay costs one round trip. As the error rate rises, replays become frequent, each one stalling the link for a round trip, and at a high enough error rate the link spends more time replaying than delivering.
Forward error correction (FEC) adds redundancy so the receiver can locate and fix errors itself, like ECC on a link. It costs bandwidth on every block, whether or not an error occurs, and it adds decoding latency to every block. In return, most errors cost nothing extra: no round trip, no stall.
Detect + replay | FEC | |
|---|---|---|
Bandwidth overhead | Small (a CRC per packet), plus a round trip per error | Fixed (parity symbols on every block) |
Latency | None normally; a round trip per error | Decoder latency on every block |
Best when | Errors are rare | Errors are frequent enough that replays would hurt |
Handles | Any error the CRC detects | Errors up to the code's correcting power; beyond that, must still detect and replay |
Link FEC usually works on symbols of several bits (for example Reed-Solomon codes with 10-bit symbols): a burst that corrupts many bits inside a few symbols costs only a few symbol corrections.
Illustrative: 256-byte packets, a 500 ns replay round trip at 32 Gb/s.
Packets per second: 32e9 / (280 x 8) ~ 14.3 million
Raw BER 1e-12: packet errors/s ~ 0.03 replay time lost: ~0.000002 %
Raw BER 1e-6: packet error probability ~ 1 - (1-1e-6)^2240 ~ 0.22%
replays per second ~ 32,000 x 500 ns = 1.6% of the time stalled
Raw BER 1e-5: packet error probability ~ 2.2% replays/s ~ 320,000 -> 16% stalled
With FEC correcting nearly all raw errors, replay is again rare; FEC costs a few % of bandwidth and tens of nsLink | Protection |
|---|---|
PCI Express Gen1 to Gen5 | CRC (LCRC) and replay only. |
PCI Express Gen6 | PAM4 signalling with fixed 256-byte flits: FEC corrects errors inside each flit, a CRC detects what FEC cannot fix, and flit-level replay recovers the rest. |
25G and faster Ethernet | Reed-Solomon FEC: RS(528,514) with 10-bit symbols (corrects up to 7 symbols) for 25G lanes, RS(544,514) (up to 15) for 50G and 100G PAM4 lanes; frames that are still bad are dropped. |
The pattern of the fastest links is all three layers: FEC fixes the common errors, a CRC catches what FEC missed or miscorrected, and replay (or a higher layer) recovers those rare cases.
A design (illustrative) added FEC to a link and removed its CRC to save bandwidth, reasoning that FEC fixed everything. Occasionally a burst exceeded the code's correcting power, the decoder miscorrected, and wrong data flowed on with no check left to catch it. FEC reduces errors; it does not replace detection.
About 2,240 × 1e-6 ≈ 0.22%.
The parity symbols on every block (bandwidth) and the decoder's latency on every block.
FEC corrects errors at the receiver: a fixed bandwidth and latency cost on every block, but no round trip per error.
Replay is cheaper while errors are rare; FEC wins once replays would stall the link noticeably.
Fast links layer FEC, a CRC behind it, and replay for what FEC cannot fix.