Last month dbglab hung under back-pressure, and someone built the wait-for table by hand from a probe, late at night. It took an hour, and the first version left out the sink, which turned out to be the arrow that closed the loop. This morning a different test hangs, in Verilator only, with the same kind of line: WATCHDOG: no bus or stream progress for 2000 cycles.
The method is known: snapshot every agent and ask what each one waits for. Can you make the table the same every time, so it takes seconds and misses nobody?
The usual start is the agent the watchdog line names. dbglab's line prints the CPU's valid, ready and address, so people start at the CPU and follow by hand: the CPU waits for the bus, the bus is held by the DMA, and so on. Three things go wrong. Agents that nobody thinks of are left out: the sink in its own clock domain, the bench's own checker. Definitions drift: is a FIFO that is full waiting, or holding? And the walk stops at the first plausible culprit, which in a cycle is any of them.
List every agent that can hold something or wait for something: masters, the fabric, slaves, FIFOs (each side), consumers in other clock domains, and software.
Write one rule per agent from the spec: a condition on signals, the thing it waits for, and the agent that can give it. A rule that cannot be written from the spec is a gap in the spec.
Give every agent a row, including the ones that wait for nothing. An idle agent is a fact, not an omission.
Take the snapshot after progress has stopped: the end of the dump that the watchdog closed, or any cycle well after the last progress.
Follow the waits from each waiting agent: back to an agent already on the path is a cycle (a deadlock); reaching agents that wait for nothing is a chain that ends (a lost credit, word or wake-up).
Check each arrow against its payload: the address and data of a waiting request, the destination of a waiting packet.
For dbglab, the rules are short enough to fit in a table. Each line is one if in the script below.
Agent | Waiting when | Waits for | Given by |
|---|---|---|---|
CPU port |
| ready (the bus) | FABRIC |
Fabric | an owner holds the grant with no request and no transfer | the owner to use or release the bus | the owner |
DMA | WAIT with | a free FIFO entry; or the bus | FIFO; or FABRIC |
FIFO | full; or empty | the sink to take a word; or a word | SINK; or CPU or DMA |
Sink |
| a CTRL write; or a word | CPU; or FIFO |
This is Observe on the debug loop, done so that it is the same every time.
A loop is a deadlock: nothing outside it can break it. A chain that ends is not a deadlock at all; something the waiting agents are owed has gone missing. The fixes are different in kind, which is why the shape comes first.
dbglab writes a VCD when it runs with DUMP=1. The watchdog ends the run, so the end of the dump is a snapshot of the hang.
$ DUMP=1 tb/run.sh dma_burst_backpressure 1
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008
$ mv dbglab.vcd b2.vcd
$ DUMP=1 tb/run.sh periph_power 1 verilator
dbglab: test=periph_power seed=1
FAIL test=periph_power seed=1 cycle=2421: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=0 cpu_ready=0 cpu_addr=00003008
$ mv dbglab.vcd b5v.vcdThe script reads only the signals the rules need, keeps their last values, and applies one rule per agent. It uses the standard library only and works on both simulators' dumps.
# not-run: reads a VCD that dbglab writes with DUMP=1 (output below is from a real run)
"""waitfor.py: the wait-for table of dbglab at a hang, read from a VCD.
usage: python3 waitfor.py <dump.vcd> [time_ns] [--edges] (default: the last time in the dump)
Each agent has one rule, written once from the design's spec: given the signal values at the
hang, what is this agent waiting for, and which agent can give it? The script prints one row per
agent, then follows the waits from every waiting agent until they loop or end."""
import sys
WANT = { # short name -> hierarchical name under dbglab_tb
"valid": "dut.m_valid", "ready": "dut.m_ready", "write": "dut.m_write", "addr": "dut.m_addr",
"grant": "dut.u_fab.grant", "busy": "dut.u_fab.busy", "lock": "dut.m_lock",
"dma": "dut.u_dma.state", "paced": "dut.u_dma.paced_q", "dreq": "dut.dreq",
"full": "dut.u_per.wfull", "empty": "dut.u_per.rempty", "sink_en": "dut.u_per.sink_en_p",
"words": "stream_words", "cycle": "cycle",
}
DMA_STATES = ["IDLE", "CHUNK", "RD_REQ", "RD_RSP", "POP", "LOAD", "WAIT", "WR_REQ", "WR_RSP", "DRAIN", "FINISH"]
def read_vcd(path, at=None):
"""Values (as bit strings) of the WANT signals at time "at" (or at the end of the dump)."""
ids, scope, vals, t = {}, [], {}, 0
with open(path) as f:
for line in f:
tok = line.split()
if not tok:
continue
if tok[0] == "$scope":
scope.append(tok[2])
elif tok[0] == "$upscope":
scope.pop()
elif tok[0] == "$var":
name = ".".join(scope + [tok[4]])
for k, h in WANT.items():
if name.endswith("dbglab_tb." + h):
ids.setdefault(tok[3], []).append(k)
elif tok[0][0] == "#":
t = int(tok[0][1:])
if at is not None and t > at * 1000: # timescale 1ps
break
elif tok[0][0] in "01xz" and tok[0][1:] in ids:
for k in ids[tok[0][1:]]:
vals[k] = tok[0][0]
elif tok[0][0] == "b" and len(tok) == 2 and tok[1] in ids:
for k in ids[tok[1]]:
vals[k] = tok[0][1:]
return vals, t // 1000
def bit(v, i, width):
"""Bit i of a VCD vector (VCD drops leading zeros)."""
v = v.rjust(width, "0")
return v[width - 1 - i]
def rules(s):
"""One row per agent: (agent, state, waits_for_agents, what_it_waits_for)."""
v, r = s["valid"], s["ready"]
cpu_v, cpu_r, dma_r = bit(v, 0, 2), bit(r, 0, 2), bit(r, 1, 2)
cpu_addr = int(s["addr"].rjust(64, "0")[32:], 2)
grant, lock = s["grant"].rjust(2, "0"), s["lock"].rjust(2, "0")
dma = DMA_STATES[int(s["dma"], 2)]
rows = []
# CPU port (master 0): a request held without ready waits for the bus.
if cpu_v == "1" and cpu_r == "0":
kind = "write" if bit(s["write"], 0, 2) == "1" else "read"
rows.append(("CPU", f"valid=1 ready=0 {kind} {cpu_addr:08x}", ["FABRIC"], "ready (the bus)"))
else:
rows.append(("CPU", f"valid={cpu_v}", [], "nothing (idle)"))
# Fabric: an owner that holds the grant without a request keeps everyone else out.
owner = {"01": "CPU", "10": "DMA"}.get(grant)
if s["busy"] == "1":
rows.append(("FABRIC", f"grant={grant} busy=1", ["SLAVE"], "a slave's response"))
elif owner and bit(v, 0 if owner == "CPU" else 1, 2) == "0":
held = "lock=" + lock
rows.append(("FABRIC", f"grant={grant} busy=0 {held}", [owner], f"its owner, the {owner}, to use or release the bus"))
else:
rows.append(("FABRIC", f"grant={grant} busy=0", [], "nothing (free or issuing)"))
# DMA engine (master 1).
if dma == "WAIT" and s["paced"] == "1" and s["dreq"] == "0":
rows.append(("DMA", f"state={dma} paced=1 dreq=0", ["FIFO"], "dreq (a free FIFO entry)"))
elif dma in ("RD_REQ", "WR_REQ") and dma_r == "0":
rows.append(("DMA", f"state={dma} ready=0", ["FABRIC"], "ready (the bus)"))
elif dma in ("RD_RSP", "WR_RSP"):
rows.append(("DMA", f"state={dma}", ["FABRIC"], "a response"))
else:
rows.append(("DMA", f"state={dma}", [], "nothing (idle)" if dma == "IDLE" else "nothing (working)"))
# Peripheral FIFO: full waits for the sink to drain; empty waits for a writer.
if s["full"] == "1":
rows.append(("FIFO", "full=1", ["SINK"], "the sink to take a word"))
elif s["empty"] == "1":
rows.append(("FIFO", "full=0 empty=1", ["CPU", "DMA"], "a word from the CPU or the DMA"))
else:
rows.append(("FIFO", "full=0 empty=0", [], "nothing (has words, has space)"))
# Sink (pclk domain): disabled waits for the CPU's CTRL write; enabled and empty waits for a word.
if s["sink_en"] == "0":
rows.append(("SINK", "sink_en=0", ["CPU"], "the CPU to set sink_en (CTRL write)"))
elif s["empty"] == "1":
rows.append(("SINK", "sink_en=1 empty=1", ["FIFO"], "a word in the FIFO"))
else:
rows.append(("SINK", "sink_en=1", [], "nothing (draining)"))
return rows
def follow(rows, start):
"""Follow the waits from "start" until they come back to an agent already on the path (a
cycle) or reach agents that wait for nothing (the chain ends: what is owed was lost)."""
waits = {a: w for a, _, w, _ in rows}
path = [start]
while True:
nxt = waits.get(path[-1], [])
back = [n for n in nxt if n in path]
if back:
return path + [back[0]], "CYCLE"
onward = [n for n in nxt if waits.get(n)]
if not onward:
return path, "ENDS at " + ", ".join(nxt) + ", which wait for nothing"
path.append(onward[0])
def main():
args = [a for a in sys.argv[1:] if a != "--edges"]
vals, t = read_vcd(args[0], int(args[1]) if len(args) > 1 else None)
missing = [k for k in WANT if k not in vals]
if missing:
sys.exit(f"signals not found in the dump: {missing}")
print(f"at {t} ns (bench cycle {int(vals['cycle'], 2)}, stream words {int(vals['words'], 2)})")
rows = rules(vals)
if "--edges" in sys.argv: # one line per wait, for cycles.py
for a, ev, w, what in rows:
for b in w:
print(f"EDGE {a} {b} {what}; {ev}")
return
print(f"{'agent':7} {'evidence':34} {'waits for':12} what")
for a, ev, w, what in rows:
print(f"{a:7} {ev:34} {','.join(w) or '-':12} {what}")
for a, _, w, _ in rows:
if w:
path, verdict = follow(rows, a)
print(f"from {a}: {' -> '.join(path)}: {verdict}")
break
if __name__ == "__main__":
main()$ python3 waitfor.py b2.vcd
at 21655 ns (bench cycle 2166, stream words 0)
agent evidence waits for what
CPU valid=1 ready=0 write 00003008 FABRIC ready (the bus)
FABRIC grant=10 busy=0 lock=10 DMA its owner, the DMA, to use or release the bus
DMA state=WAIT paced=1 dreq=0 FIFO dreq (a free FIFO entry)
FIFO full=1 SINK the sink to take a word
SINK sink_en=0 CPU the CPU to set sink_en (CTRL write)
from CPU: CPU -> FABRIC -> DMA -> FIFO -> SINK -> CPU: CYCLEThe first line is the snapshot's time: the dump closes one bus cycle after the FAIL line's count, which is why it says 2166.
Five rows, five waits: every agent is waiting, including the sink in the other clock domain, the row that was missing from last month's hand-built table.
The walk goes CPU, fabric, DMA, FIFO, sink and back to the CPU: a cycle, so a deadlock. It is the same table the case study built by hand, produced in a second from a file.
Check the edge back to the CPU: the sink waits for a CTRL write, and the CPU's waiting request is a write to 0x3008, the CTRL register. The case study's probe also showed its data, 0x05, which sets sink_en. So the arrow is real, not just plausible.
$ python3 waitfor.py b5v.vcd
at 24210 ns (bench cycle 2421, stream words 4)
agent evidence waits for what
CPU valid=0 - nothing (idle)
FABRIC grant=00 busy=0 - nothing (free or issuing)
DMA state=IDLE - nothing (idle)
FIFO full=0 empty=1 CPU,DMA a word from the CPU or the DMA
SINK sink_en=1 empty=1 FIFO a word in the FIFO
from FIFO: FIFO: ENDS at CPU, DMA, which wait for nothingThree agents wait for nothing. The CPU, the fabric and the DMA are idle: no request, no grant, no run.
The FIFO is empty and the sink is enabled, so the sink waits for a word and the FIFO waits for a writer. Both writers are idle. The walk ends.
Stream words 4. The test pushed four words before powering the sink down and six while it was down, and the bench was still waiting for those six when the watchdog fired. So six words were written into the FIFO and never came out of the stream, and the FIFO is empty. They were taken out of the FIFO by something that did not deliver them.
This is a lost-token hang, not a deadlock, and no amount of looking for a held resource would find it. The next question is where did six words go, which is a localisation job: cone-of-influence tracing follows it to the sink's read enable while its domain was off, and power-intent bugs gives the class.
The same five rules gave two different shapes on two real hangs, in two simulators, from dumps alone. On B2 the shape is a loop and every agent is part of it; the job is to find the arrow that is a design choice, which the case study does. On B5 the shape is a chain that ends at idle agents while the checker is still owed words; the job is to find where the tokens went. The table did not find either cause. It decided which kind of search comes next, in seconds, and it did not depend on who was on call.
The sink's arrow to the CPU no longer closes the loop: the sink waits for a CTRL write, but the CPU is not trying to make one. The walk would go CPU, fabric, DMA, FIFO, sink, and then the sink waits for an action the CPU is not taking, which ends the chain at the CPU's program, not at its port. You would then ask why the software did not turn the sink on, a different bug.
An empty FIFO can be given a word by either writer, so it waits for the CPU or the DMA. For it to be stuck, both must be unable to write. Here both are simply idle, so it is a chain that ends. If both writers were themselves waiting, for example for the bus, the FIFO would be stuck only if every one of those waits was stuck too.
dbglab at HEAD 49c5f04 (built with dbglab/build-history.sh), 2026-10-18. DUMP=1 tb/run.sh dma_burst_backpressure 1 (Icarus Verilog 13.0) fails with the WATCHDOG line at cycle 2165 and writes dbglab.vcd (181 kB); DUMP=1 tb/run.sh periph_power 1 verilator (Verilator 5.048) fails with WATCHDOG at cycle 2421 (240 kB).
python3 waitfor.py <dump> (Python 3.14.3, standard library only), the script shown, prints the two tables shown: the cycle CPU, FABRIC, DMA, FIFO, SINK at 21655 ns, and the chain ending at the idle CPU and DMA at 24210 ns. The B2 table matches the hand-built one in the case study.
Limits: the rules cover dbglab's five agents and read only dumped signals, so the bench's own queue of expected stream words is read from the log, not the dump. The script reads one time point; it does not check that the state was steady before it.