A new subsystem has five links: a register bus, a 64-bit stream between neighbouring blocks, a 3 mm path with four pipeline stages, a chip-to-chip link, and a DMA port into shared memory. A review asks one question for each: why this flow control and not another? "We always use valid/ready" is not an answer. This lesson turns Part 4 into a method.
Given a link, how do you choose its flow control and prove it is sized correctly?
The obvious approach is to use one mechanism everywhere, usually whichever the team knows best. It works until a link's round trip grows, or a shared resource needs protecting, or losses become possible, and then the familiar mechanism quietly costs throughput or correctness. The choice follows from a few properties of the link, and the same few questions settle it every time.
How late does the sender hear that the receiver is full? Within a cycle: valid/ready. A few cycles: valid/ready with skid buffers, or on/off with headroom. Many cycles: credits.
Can data be lost on the way? If yes, delivery needs acknowledgements and a window bounding what may need resending (Part 10), plus credits or on/off for the receiver's buffer.
Is the resource shared with traffic that must be protected? Add a rate limit (token bucket) at the sources, and separate queues or classes so one source's back-pressure does not block another's.
Is the path multi-hop? Decide hop-by-hop versus end-to-end control, switching mode, and how to keep one slow endpoint from filling shared buffers.
Link | Late by | Choice | Sizing |
|---|---|---|---|
Register bus | 0 cycles | Wait states (the target holds the transfer open) | None: one transfer at a time |
Neighbour stream, 64-bit | 0 cycles | valid/ready | None |
3 mm path, 4 stages | 4 cycles each way | valid/ready with a skid buffer per stage, or credits end to end | Skid: 2 entries per stage. Credits: B ≥ RTT |
Chip-to-chip link | Tens of cycles; bit errors possible | Credits per class, plus a replay window | B ≥ bandwidth × RTT per class; window ≥ bandwidth × ACK RTT |
DMA into shared memory | Varies | Credits or valid/ready, plus a token bucket | R = the DMA's share; S from what other clients can tolerate |
For the 3 mm path, the two options cost about the same storage. Credits keep the stages simple forward registers and concentrate the buffer at the receiver; skid buffers keep every stage self-contained. The calculation below shows both.
3 mm path: 4 forward stages, 4 return stages, 1 cycle each end (illustrative)
Skid buffers: 4 stages x 2 entries = 8 entries, spread out
Credits: RTT = 4 + 4 + 2 = 10 cycles -> B >= 10 = 10 entries at the receiver,
plus 4 plain forward registers (no ready path to time)What counts, and what frees it? Slots freed by consumption (credits), data freed by acknowledgement (window), tokens freed by time (rate).
Every term in the round trip, including return-path batching, clock-domain crossings and the receiver's own latency.
The conservation invariant, asserted every cycle in simulation, and an idle-state check that works in silicon.
Initialisation and reset: who advertises first, and what happens if one side resets alone.
Isolation: which traffic must never wait for which, and where it shares a buffer.
Lossless or lossy, and if lossless, the deadlock argument (Part 9).
Protocol | Late by | Mechanism |
|---|---|---|
APB | 0 | Wait states |
AXI, AXI-Stream | 0 per channel | valid/ready per channel |
PCI Express | Many cycles; lossy wires | Credits per type and class, plus a replay window in the data link layer |
Ethernet | Long links; congestion | Lossy by default; PAUSE or priority flow control optionally; end-to-end window in TCP |
Each protocol's choice follows the four questions. The comparison page sets more of them side by side.
A subsystem team (illustrative) standardised on valid/ready with skid buffers for every link, including one that later grew to a 20-cycle round trip after floorplanning. Each extra pipeline stage got its own skid buffer, and the link worked at full rate, but it held 40 entries of storage scattered across the route, and its ready path still had to be re-timed at every stage. Credits would have needed about 20 entries at the receiver and no ready path at all. The lesson was not that skid buffers are wrong; it was that the choice must be revisited when the round trip changes.
The mechanism and the sizing: skid buffers per stage now add up to far more storage than a single credit buffer of about 16 entries, and the ready path must be timed at every stage.
Whether data can be lost on the way. If it can, delivery needs acknowledgements and a window bounding unacknowledged data; credits alone only protect the buffer.
A rate limit (token bucket) at the DMA's port, with its burst size chosen from what the display can tolerate, plus separate queues so the DMA's back-pressure does not block display traffic.
Choose flow control from four questions: how late the stop arrives, whether data can be lost, whether the resource is shared, and whether the path is multi-hop.
Every choice ends with sizing from the round trip and an invariant to check.
Keep a per-link flow-control table in the specification, and revisit it when the round trip changes.