An on-chip network uses credits on every link, so no router buffer can ever overflow. Yet when one destination slows down, traffic for completely different destinations also slows across a large part of the network. Every link's flow control is working perfectly. The problem is what perfect per-link flow control does when one end of a path is slow.
Is flow control on every link enough, and if not, what is missing?
The obvious answer is that if no link ever overflows, the system is safe. That is true for overflow, and it is a real guarantee. But hop-by-hop control spreads a slow destination backwards: the last router's buffer fills, so it stops granting credits to the router before it, whose buffer fills, and so on. This back-pressure tree grows back towards every source sending to the slow destination, and on the way it fills buffers that other traffic needed to pass through.
Scope | Who exchanges flow-control information | Guarantees | Cannot guarantee |
|---|---|---|---|
Hop-by-hop | The two ends of each link | No buffer anywhere overflows; nothing is dropped | That a slow destination does not hold up traffic for other destinations |
End-to-end | The source and the final destination | The destination is never sent more than it can take | That intermediate buffers do not overflow (unless the network is lossy or also controlled per hop) |
The two answer different questions. Hop-by-hop control makes each link lossless. End-to-end control limits how much traffic for one destination enters the network at all, so it never piles up inside. Large lossless fabrics often use both: credits per hop to stay lossless, and an end-to-end limit (a window, or per-destination credits granted by the destination) so one slow endpoint cannot fill the fabric.
Hop-by-hop only: D slows down. Router 2's buffer for the link to D fills; Router 2 stops granting credits to Router 1 for D's traffic, and if Router 1 has one shared buffer, also for E's. Traffic to idle E stalls. Nothing is lost.
Add separate buffers per output (or virtual channels): E's traffic no longer waits behind D's in shared buffers (lessons 4.14 and 8.8). D's back-pressure still reaches Source 1, which is correct.
Add end-to-end limits: Source 1 may have only so much traffic for D in the network at once. D's slowness now holds back Source 1 at the edge, and the fabric's buffers stay mostly free for everyone else.
Approach | What it costs | When it wins |
|---|---|---|
Hop-by-hop only, shared buffers | Back-pressure spreads to unrelated traffic | Small fabrics, uniform traffic |
Hop-by-hop with per-class or per-output buffers | More buffers per router | Fabrics where some destinations are much slower than others |
End-to-end only (lossy network) | Intermediate drops and retransmission | Large networks where per-hop control is impractical |
Both | Two mechanisms to design and verify | Large lossless fabrics with uneven endpoints |
Protocol | Scope |
|---|---|
PCI Express | Hop-by-hop: credits are exchanged on each link, between the two ends of that link, so a switch has its own credits towards each neighbour. |
Ethernet with TCP | Lossy links with optional per-link PAUSE; end-to-end flow control by TCP's window between the two hosts. |
On-chip networks | Usually hop-by-hop credits per virtual channel; some add end-to-end limits for endpoints that can be slow. |
The opening slowdown (illustrative): one memory controller in a mesh network was throttled for thermal reasons. Its back-pressure filled shared router buffers along several paths, and processors talking to other, idle controllers saw their latency rise several times. Every link's credits were correct. The fix was separate virtual channels for traffic to different memory controllers, plus a cap on outstanding requests per destination at each source, an end-to-end limit.
No: every link's receiver has room for everything its sender is allowed to send. It is lossless, which is exactly why back-pressure spreads instead of packets being dropped.
It bounds how much traffic for one destination can be inside the network, so a slow destination holds back its own senders at the edge instead of filling shared buffers.
Separate buffers (or virtual channels) per output or per class, so the idle destination's traffic does not queue behind the slow one's.
Hop-by-hop flow control guarantees no buffer overflows, but a slow endpoint's back-pressure spreads through shared buffers to unrelated traffic.
End-to-end control limits how much traffic for one destination enters the network, but does not protect intermediate buffers on its own.
Large lossless fabrics often combine hop-by-hop credits, separate buffers per class, and end-to-end limits.