A storage controller sends data across a PCI Express switch to a host. Every link checks every packet with a CRC and resends any that fail, so the team is confident nothing can be corrupted. Then a rare bug inside the switch flips a bit in a packet after it passed the incoming link's check and before the outgoing link computed its new CRC. Both links report a clean transfer. The host receives wrong data with a valid CRC.
If every hop is protected, how can data still arrive wrong, and which layer should have caught it?
The obvious answer is to protect every hop: if each link delivers exactly what it was given, the chain delivers exactly what the source sent. The flaw is the word "given". Each link protects only the wire it owns. Anything that happens between links, inside a switch, a bridge, a buffer or a clock-crossing FIFO, is outside every link's check. A chain of perfect links can still deliver corrupted data, because the corruption happened where no link was looking.
Real protocols are built as a stack of layers. Each layer offers a service to the layer above and uses the service of the layer below, and each sees only its own header and trailer wrapped around what the layer above handed it, which is called encapsulation. A typical stack has three layers:
Layer | Promises to the layer above | Typical mechanisms |
|---|---|---|
Transaction | Reads, writes and messages with the right meaning and ordering | Addresses, tags, ordering rules, completions |
Link (data link) | Packets delivered across one hop, intact, in order, without overflowing the receiver | Sequence numbers, per-hop CRC, replay, credits |
Physical | Bits (or symbols) across one wire or lane | Clocking, encoding, framing, lane alignment, training |
Layering lets each layer be designed, replaced and verified on its own: the physical layer can move to a faster encoding without the transaction layer noticing. But it raises a question every protocol designer has to answer: which layer owns a guarantee such as "the data arrives correct"?
A classic paper on system design (Saltzer, Reed and Clark, 1984) gives the answer. Paraphrased: if a function can be done correctly only with knowledge that exists at the end points, it must be done at the end points. A lower layer may also do it, but only as a performance enhancement, never as the guarantee.
Correct delivery is such a function. Only the source knows what it meant to send, and only the final destination knows what it received. Every intermediate layer sees a copy, and every copy can be damaged somewhere no hop protects. So a guarantee of correct end-to-end delivery needs an end-to-end check. Per-hop checks are still valuable, because they catch most errors early and cheaply, close to where they happen, so the end-to-end mechanism rarely has to act. They make the system fast; they cannot make it correct.
Apply the argument to four things a system wants, and see where each must live:
Guarantee | Who has the knowledge? | Must live at | Lower layers may add, for speed |
|---|---|---|---|
Data arrives uncorrupted | Source and destination only | End to end (an end-to-end CRC or checksum) | Per-hop CRC and replay, so errors are fixed close to where they happen |
Each hop does not overflow its receiver | The two ends of that hop | The link layer (credits, Part 4) | Nothing: the hop is the end point here |
Writes become visible in program order | The requester and its final target | The transaction layer, end to end (Part 7) | Fabric ordering rules that preserve it hop by hop |
A request gets an answer | The requester | The requester (timeouts, Part 9) | Per-hop error responses that answer early |
The second row shows the argument is not "do everything end to end". Flow control on a hop is an end-to-end function of that hop, so the link layer is the right owner. The skill is asking, for each guarantee: whose knowledge does it need?
Where to put a check | What it costs | When it wins |
|---|---|---|
Per hop only | Cheap; catches wire errors near their source | Systems where nothing between hops can corrupt data (rare) or where an end-to-end check exists elsewhere |
End to end only | Every error travels all the way before it is caught; recovery is slow and far away | Low error rates and simple fabrics |
Both | Two checks' worth of logic and bits | High-reliability systems: per hop for speed, end to end for correctness |
Protocol | How it layers protection |
|---|---|
PCI Express | Three layers: transaction, data link and physical. The data link layer adds a CRC (LCRC) on every hop and replays packets that fail it; a transaction-layer packet may also carry an optional end-to-end CRC (ECRC) that switches pass through untouched. |
Ethernet with TCP | Ethernet checks each frame with a CRC and drops bad ones; TCP, at the end points, checks and retransmits. The link catches errors early; the end points guarantee delivery. |
AXI | One layer on chip. It defines no retransmission; a failed transfer gets an error response, and data protection, if needed, is added around it (parity or ECC, Part 10). |
The opening story is the end-to-end argument learned the hard way (illustrative). The team had enabled per-hop CRC everywhere and left the optional end-to-end CRC off to save logic. The switch's internal buffer had a rare soft-error path that was not covered by parity. A single flipped bit inside the switch was recomputed into a fresh, valid CRC on the outgoing link.
Symptom: occasional wrong data at the host, with no error logged anywhere.
Evidence: every link's error counters were zero; the corruption correlated with traffic through one switch.
Root cause: the only checks were per hop; the switch's internals were between them.
Fix: enable the end-to-end CRC, and add parity on the switch's internal buffers so errors there are caught early too.
Can anything between two link checks corrupt data: buffers, FIFOs, clock crossings, bridges? If any such element exists without its own protection, per-hop CRCs cannot catch errors there, and an end-to-end check is needed.
Its job is to keep one hop's receiver from overflowing, and the only parties with the knowledge it needs are the two ends of that hop. For that function the hop is the end-to-end path.
It lets each layer be designed and changed independently, because each sees only its own header and trailer. It hides the lower layers' state from the upper ones, which is exactly why an upper layer cannot rely on a lower layer for guarantees it cannot see.
Protocols are stacked in layers; each wraps the unit from above in its own header and trailer and promises a service to the layer above.
End-to-end argument: a guarantee that needs knowledge only the ends have must be implemented at the ends; lower layers may add it only for speed.
Per-hop protection catches errors early; only an end-to-end check covers what happens between hops.