A small ring interconnect links four hosts. Each node keeps the packets passing through it in a two-entry buffer, and each host injects its own packets into its node. Last week someone changed one rule: when a host wants to inject and a packet is passing, the host now wins, to give busy hosts a fair share. This week the ring hangs in every stress test, and the ticket says host 2 cannot inject. The host team is asked to look.
Host 2 is certainly stuck. Is it the cause?
The usual fix starts from the agent in the ticket: give host 2 priority, deepen its buffer, add a retry. Each of these changes the victim and leaves the cause alone. In a deadlock most stuck agents are not part of the cycle; they wait for something inside it. Fix one of them and the hang comes back with a different name in the ticket. Reading the graph by eye has the same problem at scale: a person picks the path they started from.
The previous lesson turned a snapshot into a table and followed one path. This one treats the whole table as a graph and searches it.
Dump the graph at the hang automatically. One line per wait: waiter, waited-for, and the reason in the design's words. The bench prints it when the watchdog fires; on silicon, the hang detector's snapshot carries it.
Find the strongly connected components of the waiting agents. A component with two or more members (or one agent waiting for itself) is a cycle. Tarjan's algorithm does it in one pass.
Classify everyone else: blocked behind a cycle if its waits lead into one; part of a chain that ends if its waits lead only to agents that wait for nothing.
Read the reasons along the cycle and find the edge that is a design choice rather than physics. A full buffer waiting for the next is physics; who gets the free entry is a choice.
Move from the dynamic graph to the static one. The dependency buf0 to buf1 to buf2 to buf3 exists in every ring; what matters is whether the state that fills it can be reached. A fix must make that state unreachable, not just rarer.
Check the state, not the symptom: an invariant that fails the moment the cycle's state forms (here, every buffer full), run across seeds and loads.
This is Localise on the debug loop: from "everything is stuck" to the four agents and one rule where the bug lives.
FIX=1 is the original rule, transit first: a packet already on the ring wins the free entry. FIX=0 is last week's change, local first. The bench offers packets at +LOAD percent, and when nothing has moved for 50 cycles it prints one EDGE line per wait.
// ring4: four nodes on a one-way ring (0 -> 1 -> 2 -> 3 -> 0). Node i keeps the packets that are
// at node i in a two-entry buffer. Each cycle the head packet of every buffer either leaves the
// ring (it has arrived) or moves to the next node's buffer if that buffer has a free entry.
// A host at each node injects new packets into its own node's buffer.
// FIX=1: a packet already on the ring wins over an injection (transit first).
// FIX=0: the host wins (local first), a change made to give busy hosts a fair share.
module ring4 #(parameter bit FIX = 1) (
input logic clk, rst_n,
input logic [3:0] inj_valid, // host i has a packet for the ring
input logic [7:0] inj_dest, // 2 bits per host: destination node
output logic [3:0] inj_ready, // host i's packet is taken this cycle
output logic [3:0] eject // a packet left the ring at node i
);
logic [1:0] dest [4][2]; // buffer i, entry 0 is the head
logic [1:0] cnt [4];
logic [3:0] move, arrive;
always_comb begin
for (int i = 0; i < 4; i++) begin
int nx;
nx = (i + 1) % 4;
eject[i] = (cnt[i] != 0) && (dest[i][0] == 2'(i));
move[i] = (cnt[i] != 0) && (dest[i][0] != 2'(i)) && (cnt[nx] < 2'd2)
&& (FIX || !(inj_valid[nx])); // local first: the host takes the free entry
end
for (int i = 0; i < 4; i++) begin
arrive[i] = move[(i + 3) % 4];
inj_ready[i] = inj_valid[i] && !arrive[i] && (cnt[i] < 2'd2);
end
end
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) for (int i = 0; i < 4; i++) cnt[i] <= '0;
else for (int i = 0; i < 4; i++) begin
logic [1:0] c;
logic pop;
pop = eject[i] || move[i];
c = cnt[i];
if (pop) begin dest[i][0] <= dest[i][1]; c = c - 2'd1; end
if (arrive[i] || inj_ready[i]) begin
dest[i][c[0]] <= arrive[i] ? dest[(i + 3) % 4][0] : inj_dest[2*i +: 2];
c = c + 2'd1;
end
cnt[i] <= c;
end
endmodule
$ iverilog -g2012 -o r.vvp ring_tb.sv && vvp -n r.vvp
PASS: 800 packets delivered in 820 cycles (FIX=1)
$ iverilog -g2012 -Pring_tb.FIX=0 -o rb.vvp ring_tb.sv && vvp -n rb.vvp +LOAD=30
HANG at cycle 107: sent 12/14/9/13, delivered 40
EDGE buf0 buf1 head for node 2 needs a free entry in buf1 (full)
EDGE host0 buf0 new packet needs a free entry in buf0 (count 2)
EDGE buf1 buf2 head for node 3 needs a free entry in buf2 (full)
EDGE host1 buf1 new packet needs a free entry in buf1 (count 2)
EDGE buf2 buf3 head for node 0 needs a free entry in buf3 (full)
EDGE host2 buf2 new packet needs a free entry in buf2 (count 2)
EDGE buf3 buf0 head for node 0 needs a free entry in buf0 (full)
EDGE host3 buf3 new packet needs a free entry in buf3 (count 2)
FAIL cycle=107: no packet moved for 50 cyclesEven at 30% load the changed ring hangs after 40 packets. Every one of seeds 1 to 20 hangs, between cycles 56 and 135. Eight lines are easy to read by eye; a ring of 64 nodes with four virtual channels would print hundreds. So search it.
The script reads EDGE lines, runs Tarjan's algorithm, and reports cycles, the agents behind each cycle, and chains that end. Run with no argument, it checks itself on the dump above.
"""cycles.py: find the cycles in a wait-for graph dumped at a hang.
usage: python3 cycles.py <edges.txt | -> (no argument: the ring4 dump below, with a self-check)
Input lines: EDGE <waiter> <waited-for> <reason...>; other lines are ignored.
Prints every cycle (a strongly connected group of waiters), the agents blocked behind a cycle,
and the chains that end at an agent waiting for nothing (a lost event, not a deadlock)."""
import sys
RING4_SEED1 = """\
HANG at cycle 107: sent 12/14/9/13, delivered 40
EDGE buf0 buf1 head for node 2 needs a free entry in buf1 (full)
EDGE host0 buf0 new packet needs a free entry in buf0 (count 2)
EDGE buf1 buf2 head for node 3 needs a free entry in buf2 (full)
EDGE host1 buf1 new packet needs a free entry in buf1 (count 2)
EDGE buf2 buf3 head for node 0 needs a free entry in buf3 (full)
EDGE host2 buf2 new packet needs a free entry in buf2 (count 2)
EDGE buf3 buf0 head for node 0 needs a free entry in buf0 (full)
EDGE host3 buf3 new packet needs a free entry in buf3 (count 2)
"""
def parse(text):
edges = {} # waiter -> [(waited_for, reason)]
for line in text.splitlines():
tok = line.split(maxsplit=3)
if len(tok) >= 3 and tok[0] == "EDGE":
edges.setdefault(tok[1], []).append((tok[2], tok[3] if len(tok) > 3 else ""))
return edges
def sccs(edges):
"""Tarjan's algorithm: groups of agents that can all reach each other."""
nodes = sorted(set(edges) | {b for outs in edges.values() for b, _ in outs})
index, low, on, stack, out, n = {}, {}, set(), [], [], [0]
def visit(v):
index[v] = low[v] = n[0]
n[0] += 1
stack.append(v)
on.add(v)
for w, _ in edges.get(v, []):
if w not in index:
visit(w)
low[v] = min(low[v], low[w])
elif w in on:
low[v] = min(low[v], index[w])
if low[v] == index[v]:
group = []
while True:
w = stack.pop()
on.discard(w)
group.append(w)
if w == v:
break
out.append(group)
for v in nodes:
if v not in index:
visit(v)
return nodes, out
def cycle_order(group, edges):
"""Walk one cycle inside a strongly connected group, starting at its first name."""
start = min(group)
path = [start]
while True:
nxt = next(w for w, _ in edges[path[-1]] if w in group)
if nxt == start:
return path + [start]
if nxt in path: # a smaller loop inside the group
return path[path.index(nxt):] + [nxt]
path.append(nxt)
def analyse(edges):
nodes, groups = sccs(edges)
cycles = [g for g in groups if len(g) > 1 or any(w == g[0] for w, _ in edges.get(g[0], []))]
in_cycle = {v: i for i, g in enumerate(cycles) for v in g}
reach = {} # agent -> the cycle it is stuck behind, if any
def behind(v, seen=()):
if v in in_cycle:
return in_cycle[v]
if v in reach:
return reach[v]
for w, _ in edges.get(v, []):
if w not in seen:
c = behind(w, seen + (v,))
if c is not None:
reach[v] = c
return c
reach[v] = None
return None
report = {"cycles": [], "behind": {}, "dead_ends": []} # dead_ends: waiters behind no cycle
for i, g in enumerate(cycles):
report["cycles"].append(cycle_order(g, edges))
for v in nodes:
if v in in_cycle or v not in edges:
continue
c = behind(v)
if c is not None:
report["behind"].setdefault(c, []).append(v)
else:
report["dead_ends"].append(v)
return report
def show(edges, report):
print(f"{sum(len(o) for o in edges.values())} waits between {len(set(edges) | {b for o in edges.values() for b, _ in o})} agents")
for i, path in enumerate(report["cycles"], 1):
print(f"cycle {i} ({len(path) - 1} agents): {' -> '.join(path)}")
for a, b in zip(path, path[1:]):
print(f" {a} waits for {b}: " + next(r for w, r in edges[a] if w == b))
if report["behind"].get(i - 1):
print(f" blocked behind it: {', '.join(report['behind'][i - 1])}")
waited = {b for o in edges.values() for b, _ in o}
for v in report["dead_ends"]:
if v in waited:
continue # printed as part of a longer chain
path = [v]
while True:
onward = [w for w, _ in edges[path[-1]] if w in edges and w not in path]
if not onward:
break
path.append(onward[0])
ends = [w for w, _ in edges[path[-1]] if w not in edges]
print(f"chain ends: {' -> '.join(path)} -> {' or '.join(ends)}, which wait for nothing (a lost event?)")
if not report["cycles"]:
print("no cycle: not a deadlock")
def main():
if len(sys.argv) > 1:
text = sys.stdin.read() if sys.argv[1] == "-" else open(sys.argv[1]).read()
edges = parse(text)
show(edges, analyse(edges))
return
edges = parse(RING4_SEED1)
report = analyse(edges)
show(edges, report)
assert report["cycles"] == [["buf0", "buf1", "buf2", "buf3", "buf0"]], report
assert sorted(report["behind"][0]) == ["host0", "host1", "host2", "host3"], report
print("PASS: one 4-agent cycle, 4 hosts blocked behind it")
if __name__ == "__main__":
main()$ vvp -n rb.vvp +LOAD=30 | python3 cycles.py -
8 waits between 8 agents
cycle 1 (4 agents): buf0 -> buf1 -> buf2 -> buf3 -> buf0
buf0 waits for buf1: head for node 2 needs a free entry in buf1 (full)
buf1 waits for buf2: head for node 3 needs a free entry in buf2 (full)
buf2 waits for buf3: head for node 0 needs a free entry in buf3 (full)
buf3 waits for buf0: head for node 0 needs a free entry in buf0 (full)
blocked behind it: host0, host1, host2, host3One cycle, four members: the four buffers, each full, each head bound for a later node.
Four victims: every host, host 2 included. None of them is on the cycle, so no change to a host can break it.
The reasons are all physics: a full buffer waiting for a free entry in the next. The choice is not on any edge; it is in how the ring got full. That is the static question.
The previous lesson's waitfor.py prints its table as EDGE lines with --edges, so the same search runs on dbglab's two real hangs:
$ python3 waitfor.py b2.vcd --edges | python3 cycles.py -
5 waits between 5 agents
cycle 1 (5 agents): CPU -> FABRIC -> DMA -> FIFO -> SINK -> CPU
CPU waits for FABRIC: ready (the bus); valid=1 ready=0 write 00003008
FABRIC waits for DMA: its owner, the DMA, to use or release the bus; grant=10 busy=0 lock=10
DMA waits for FIFO: dreq (a free FIFO entry); state=WAIT paced=1 dreq=0
FIFO waits for SINK: the sink to take a word; full=1
SINK waits for CPU: the CPU to set sink_en (CTRL write); sink_en=0
$ python3 waitfor.py b5v.vcd --edges | python3 cycles.py -
3 waits between 4 agents
chain ends: SINK -> FIFO -> CPU or DMA, which wait for nothing (a lost event?)
no cycle: not a deadlockB2 is a five-agent cycle with no victims, and here one edge is a choice: the fabric keeping the grant for an owner that waits, which is what the case study fixes. B5 has no cycle at all. One script, three systems, three shapes.
The static dependency graph of any one-way ring has the cycle buf0, buf1, buf2, buf3: a packet in one buffer can always need the next. So the cycle cannot be designed away; the question is whether the state that closes it, every buffer full, can be reached. The bench's +FULLCHK fails the moment it is:
$ vvp -n rb.vvp +LOAD=30 +FULLCHK
FAIL cycle=57: RING_FULL: every buffer is full, no packet can moveThe trap closes at cycle 57; the watchdog reported it 50 cycles later. Now the argument for transit first. A host at node j may inject only when j has a free entry and nothing arrives at j. So the head packet at node j-1, if there is one, is neither moving into j nor blocked by j: it must be leaving the ring. Every injection is therefore paid for by an ejection at the node behind it, or happens behind an empty node, which gains at most one packet in a cycle (an arrival and an injection never happen together) and so cannot be full one cycle later. To go from not full to full in one cycle the ring needs more injections than ejections, so at least one injection sits behind an empty node, and that node is still not full: the ring never fills. And a ring that is not full always has a head that can move or leave. Local first breaks the first step: the host injects while the packet behind it waits, so an injection no longer costs an ejection.
The argument covers every run; the runs check it. With FIX=1 and +FULLCHK, seeds 1 to 20 at 100% load all pass, so the full state never appeared even with every host injecting whenever it could.
The search named the four buffers as the cycle and the four hosts, including host 2, as victims. The reasons on the cycle's edges showed no choice, which sent the search to the static question: how the ring fills. One rule decides that, and the bug was the change to it. The invariant check fires at the cause, 50 cycles before the watchdog, and the same script found the cycle in dbglab and correctly found none in a lost-token hang.
Whether any of the five is on the three-agent cycle. If none is, doubling their buffers cannot break the deadlock; it only delays the moment they fill. Look at the reasons on the three cycle edges and find the one that is a choice, or the rule that lets that state form.
+FULLCHK at 100% load, while the changed ring trips it at cycle 57 even at 30%?With transit first, the last free entry can only be taken by a packet that frees another entry in the same cycle, so the ring never fills, whatever the load. With local first, a host can take the last free entry while the packet behind it waits, and that one move closes the cycle. Load changes how often the chance arises, not whether it can.
ring_tb.sv as shown, 2026-10-18, Icarus Verilog 13.0: iverilog -g2012 -o r.vvp ring_tb.sv && vvp -n r.vvp PASS (820 cycles); with -Pring_tb.FIX=0, vvp -n rb.vvp +LOAD=30 HANG at cycle 107 with eight EDGE lines, +FULLCHK RING_FULL at cycle 57. Seeds 1 to 20 at load 30 all hang (cycles 56 to 135); loads 30, 50, 70 and 90, seeds 1 to 3, all hang. The fixed ring passes seeds 1 to 3 at those loads, and seeds 1 to 20 at load 100 with +FULLCHK. Verilator 5.048 prints the same HANG line.
cycles.py as shown (Python 3.14.3): with no argument it analyses the embedded seed-1 dump and prints PASS; vvp -n rb.vvp +LOAD=30 | python3 cycles.py - gives the same report. On dbglab HEAD 49c5f04, python3 waitfor.py <dump> --edges | python3 cycles.py - (the previous lesson's script) finds the five-agent cycle in B2 and the chain ending at the CPU or DMA in B5. vvp -n r.vvp +LOAD=90 +SNAP=200 | python3 cycles.py - reports four waits and no cycle.
Limits: a four-node ring with single-flit packets and an atomic one-hop move, simulated only. The proof sketch for transit-first is an argument, not a formal proof; a formal tool would check it on the RTL.