An on-chip network never drops a packet: if a buffer is full, the sender waits. A data-centre Ethernet switch, faced with the same full buffer, drops the frame and lets the endpoints sort it out. Both are well-engineered. They made opposite choices for the same moment, and each choice brings its own failures.
When a buffer is full, should the network stop the sender or drop the packet?
Dropping data sounds like a bug, so the obvious answer is never to drop: make every link lossless. But a lossless network replaces lost packets with waiting, and waiting spreads. The back-pressure tree of lesson 4.11 can reach across a whole network, and if the waits form a cycle, nothing moves at all. Dropping is crude, but it keeps a congested part of the network from freezing the rest. Neither choice is free.
Lossless | Lossy | |
|---|---|---|
When the buffer is full | The sender is stopped (credits, ready, PAUSE) | The packet is dropped |
Who recovers | Nobody needs to: nothing was lost | The endpoints, by detecting the loss and resending (Part 10) |
Latency | Low and predictable while uncongested; waits spread under congestion | Low while uncongested; a loss costs a timeout or a round trip |
Congestion spreads? | Yes, as back-pressure through shared buffers | No: the congested buffer drops and stays local |
Deadlock possible? | Yes, if buffer waits form a cycle (Part 9) | No buffer deadlock: drops break every cycle |
Needs | Per-hop flow control, careful buffer and channel design | Reliable delivery at the ends, and tolerance of jitter |
On chip, losing a packet is unaffordable. A dropped cache-line read or write must be recovered by hardware that does not otherwise exist; the round trips are a few cycles, so per-hop credits are cheap; and the topology is fixed, so deadlock can be designed out. On-chip networks are lossless.
Across a large network, per-hop control is costly and congestion is common. Links are long, topologies change, and many independent flows share switches. Dropping keeps congestion local, and the endpoints already need reliability for other reasons (the end-to-end argument, lesson 0.7). Classic Ethernet is lossy.
Some networks need both properties. Storage and remote-memory traffic hates losses (a resend costs far more than the wait), so those fabrics are built lossless even when they are large, and then must manage the costs.
Protocol or network | Choice |
|---|---|
On-chip networks, PCI Express | Lossless: per-link credits mean a packet is never sent without room for it. (PCIe's replay mechanism recovers from bit errors, not from full buffers.) |
InfiniBand, Fibre Channel | Lossless: credit-based link-level flow control. |
Classic Ethernet | Lossy: a switch with a full buffer drops the frame, and frames with a bad check sequence are dropped; TCP at the ends recovers. |
Ethernet for RDMA (RoCEv2) | Made lossless with priority flow control (PAUSE per traffic class), which brought pause storms and deadlocks at data-centre scale. |
Lossless: congestion spreading and deadlock. When Ethernet for remote memory access was made lossless with priority flow control, operators at data-centre scale saw a single misbehaving receiver send PAUSE frames that spread across many switches (a pause storm), and cyclic buffer dependencies that froze traffic entirely (a PFC deadlock). These are exactly the failures a lossless network trades drops for.
Lossy: losses that are too expensive. A lossy network carrying traffic whose endpoints recover slowly, with long timeouts, turns each drop into a stall of many round trips. If loss is rare, that is fine; under steady congestion it can collapse throughput.
In a lossless network a full buffer makes its upstream wait; if the waits form a cycle, no buffer can drain. A lossy network drops instead of waiting, which breaks every such cycle.
No. Credits ensure a packet is never sent without room, so full buffers never cause loss. Replay recovers from bit errors on the wire, a different kind of loss.
Reliable delivery: detecting the loss (sequence numbers, timeouts) and resending, with the recovery time showing up in the latency tail.
A full buffer either stops the sender (lossless) or drops the packet (lossy).
Lossless avoids recovery but spreads congestion and can deadlock; lossy keeps congestion local but needs recovery at the ends.
On-chip networks and PCIe are lossless; classic Ethernet is lossy; lossless Ethernet for RDMA inherits pause storms and deadlock.