A system-on-chip hangs about once a day under a stress test. The waveform dump from the moment the watchdog fired has 40,000 signals. Somewhere in it is the reason. Scrolling through it hoping to spot something is how such bugs take weeks. A method finds it in an afternoon: at the frozen moment, ask every agent what it holds and what it is waiting for, and draw the arrows.
How do you find the cause of a hang from a waveform?
The obvious approach is to work backwards from the first symptom: the processor stopped, so look at the processor's last request, then whatever it went to, and so on. That follows one thread through a tangle, and the cause of a deadlock is not on one thread: it is a loop through several agents, each of which looks merely "waiting". Hang debug needs the whole picture at one moment, organised as a graph.
The last branch matters: a hang with no cycle and no activity is usually a lost event (a response that was dropped, a credit that leaked (lesson 4.8), an interrupt that fired while masked). The wait-for graph then ends at an agent waiting for something that no one owes it.
Here is a small design that hangs, with a watchdog that fires after 8 idle cycles and prints the state. Before reading the code's comments, look at the waveform and apply the method.
// A design that hangs, for debugging. A requester and a target follow AXI-style
// write channels: an address channel (aw_valid / aw_ready) and a data channel
// (w_valid / w_ready). Each side looks reasonable on its own:
// requester: sends the address, and sends the data only after the address has
// been accepted (this breaks the rule that a requester must not wait
// for the address to be accepted before offering data)
// target: accepts the address only once it can see the data (allowed)
// A watchdog counts cycles without any handshake and, when it fires, prints the
// wait-for state: who is waiting for what. The testbench passes when the watchdog
// catches the hang and the state it reports forms a cycle.
module tb;
logic clk = 0;
always #5 clk = ~clk;
initial begin $dumpfile("wave.vcd"); $dumpvars(0, tb); end
int cyc = 0;
always @(posedge clk) cyc <= cyc + 1;
logic aw_valid = 0;
logic w_valid = 0; // requester: data only after the address was accepted
// Target
wire aw_ready = w_valid; // address accepted only when data is visible
wire w_ready = 1;
// Requester
always @(posedge clk) begin
if (cyc == 1) aw_valid <= 1; // start a write in cycle 1
if (aw_valid && aw_ready) begin aw_valid <= 0; w_valid <= 1; end
end
// Watchdog: cycles since the last handshake on either channel
int idle = 0;
logic hang = 0;
always @(posedge clk) begin
if ((aw_valid && aw_ready) || (w_valid && w_ready)) idle <= 0;
else if (aw_valid || w_valid || cyc >= 2) idle <= idle + 1;
if (idle == 8 && !hang) begin
hang <= 1;
$display("watchdog: no handshake for 8 cycles. aw_valid=%0b aw_ready=%0b w_valid=%0b", aw_valid, aw_ready, w_valid);
$display(" requester waits for aw_ready before w_valid; target waits for w_valid before aw_ready: a cycle");
end
end
initial begin
#200;
if (hang && aw_valid && !aw_ready && !w_valid) $display("PASS: the watchdog caught the hang; the wait-for state is a cycle between aw_ready and w_valid");
else $display("FAIL: hang %0b aw_valid %0b aw_ready %0b w_valid %0b", hang, aw_valid, aw_ready, w_valid);
$finish;
end
endmoduleApply the method at slot 9, after progress has stopped:
Channels. Address channel: aw_valid 1, aw_ready 0: an address is offered and not accepted. Data channel: w_valid 0, w_ready 1: the target would take data, but none is offered.
Agents. The requester holds an address on offer and is waiting for aw_ready before it offers data. The target is waiting for w_valid before it raises aw_ready.
Arrows. requester's data → aw_ready; aw_ready → w_valid; w_valid is the requester's data. A cycle.
Which arrow is illegal? The target waiting for data before accepting the address is allowed. The requester waiting for the address to be accepted before offering data is forbidden by the dependency rule (lesson 9.3). Fix the requester.
The watchdog's printed lines say the same thing in words; on a real design, a watchdog that dumps the state of every channel turns step 1 from hours into seconds.
Feature | Why |
|---|---|
Watchdogs and progress monitors that dump channel state | The frozen moment is captured automatically |
Per-agent counters: requests issued, responses received, credits held, tags in use | A leak or a lost event shows as a counter that never returns to its idle value |
Readable status for every queue and tracker in silicon | The same method works on a chip in the lab, through a debug port |
Protocol checkers on every interface in simulation | Most illegal arrows are caught when they first happen, before they become hangs |
The once-a-day hang (illustrative) was a credit leak: the wait-for graph at the frozen moment had no cycle, and nothing was moving. It ended at a sender waiting for a credit that no receiver owed it: the receiver's buffer was empty and its credit counter was at the limit, but the sender's count was two short. Following the counters back led to a credit dropped when a return coincided with a reset of one side.
w_ready is 1 throughout: the data channel is idle only because w_valid is 0.
A lost event: a dropped response, a leaked credit, or a wake-up or interrupt that was missed, so someone waits for something no one owes it.
Debug hangs at one frozen moment: list every channel and every agent's holdings and waits, and draw the wait-for graph.
A cycle is a deadlock; activity without completion is livelock or starvation; no cycle and no activity is a lost event.
Design for debug: state-dumping watchdogs, leak-revealing counters, and protocol checkers on every interface.