A product in the field hangs about once a week. Its system watchdog does its job: software stops kicking it, it resets the chip, and the unit comes back after two seconds. The field logs say watchdog reset and nothing else. Nobody can reproduce it in the lab, and the simulation regression is green.
If you could add one block to the next chip, what would it have to record, and why must it survive the very reset that recovers the hang?
Most chips have one of two things. A system watchdog recovers but explains nothing: by the time software can look, the reset has cleared every state machine and pointer. A bus timeout that answers a stuck request with an error is better, because the CPU at least learns which access failed, but it only covers the one master that is waiting on the bus. Neither says which agent stopped first or what the system looked like when it did.
The bench guards from Part 1 do not exist on silicon, and a single "no bus activity" detector built into the design falls into the same polling trap: a CPU spinning on a status register looks busy for ever.
Pick the ports that can wait: every master interface, every queue with a consumer in another block, every engine with a done signal.
Define pending and progress for each, as in hang, livelock or starvation: pending is "waiting for something", progress is "got it". A port that is idle is not pending.
Time only the wait: a counter that runs while pending and not progressing, and clears on progress. A shared prescaler keeps the counters a few bits wide.
Make the limit a register, set from measured legal waits with margin, and give it a pause input for low-power phases.
Keep the first hang: sticky bits per port, plus the first port, a timestamp and a snapshot of the state that answers "who waits for whom" (grants, locks, FSM states, FIFO flags). Later hangs set bits and nothing else.
Offer recovery, keep evidence: a reset request output for the system, and a detector reset that only power-on drives. Read it through a side-band path, as observability by design argues.
Test it like logic: legal slow waits must pass, a stuck port must fire with the right record, the record must survive the functional reset, and a second hang must not overwrite it.
On the debug loop this is Observe, built in at Prevent time: the next hang arrives with its own snapshot.
hang_det below has four ports, a prescaler of 8 cycles and an 8-bit timer per port. The bench drives four toy requesters whose answers come after a chosen delay, or never.
// hang_det: a hang detector to build into a design. Each of N ports reports pend (it is waiting:
// a request is outstanding) and prog (it made progress this cycle). A shared prescaler ticks
// every 2**PW cycles; a port's timer counts ticks while it is pending with no progress and clears
// on progress or when nothing is pending. A timer that reaches limit_i sets the port's sticky
// hung bit. The first hang also records the port, a timestamp and a snapshot of snap_i, and
// pulses rst_req_o (a request for a functional reset, if the integrator wants recovery).
// The detector has its own reset (dbg_rst_n), so its record survives the functional reset it
// asks for. clr_i clears the record and re-arms it; en_i = 0 pauses all timers (low power).
module hang_det #(parameter int N = 4, parameter int PW = 3, parameter int TW = 8, parameter int SW = 16) (
input logic clk, dbg_rst_n,
input logic en_i, clr_i,
input logic [TW-1:0] limit_i, // in ticks of 2**PW cycles
input logic [N-1:0] pend_i, prog_i,
input logic [SW-1:0] snap_i,
output logic [N-1:0] hung_o, // sticky, one bit per port
output logic [$clog2(N)-1:0] first_o, // the port that hung first
output logic [31:0] first_ts_o, // and when, in cycles since dbg_rst_n
output logic [SW-1:0] snap_o, // the state at that moment
output logic rst_req_o
);
logic [PW-1:0] pre;
logic [TW-1:0] t [N];
logic [31:0] ts;
logic tick;
assign tick = en_i && (pre == '0);
always_ff @(posedge clk or negedge dbg_rst_n)
if (!dbg_rst_n) begin
pre <= '0; ts <= '0; hung_o <= '0; first_o <= '0; first_ts_o <= '0; snap_o <= '0; rst_req_o <= 1'b0;
for (int i = 0; i < N; i++) t[i] <= '0;
end else begin
logic [N-1:0] fire;
pre <= pre + 1'b1;
ts <= ts + 32'd1;
rst_req_o <= 1'b0;
for (int i = 0; i < N; i++) begin
fire[i] = 1'b0;
if (!pend_i[i] || prog_i[i] || clr_i) t[i] <= '0;
else if (tick && t[i] != limit_i) t[i] <= t[i] + 1'b1;
if (pend_i[i] && !prog_i[i] && t[i] == limit_i && !hung_o[i]) fire[i] = 1'b1;
end
if (clr_i) hung_o <= '0;
else begin
hung_o <= hung_o | fire;
if (hung_o == '0 && fire != '0) begin // the first hang only
for (int i = N - 1; i >= 0; i--) if (fire[i]) first_o <= $clog2(N)'(i);
first_ts_o <= ts;
snap_o <= snap_i;
rst_req_o <= 1'b1;
end
end
end
endmodule
$ iverilog -g2012 -o h.vvp hang_det_tb.sv && vvp -n h.vvp
cycle 172: rst_req: hung=0100 first=2 ts=169 snap=44ab
after the functional reset: hung=0100 first=2 ts=169 snap=44ab
after a second hang: hung=0110 first=2 ts=169 snap=44ab
PASS: slow answers ignored, hang caught with port and snapshot, record kept through resetThe legal answers passed silently: 10, 25 and 40 cycles against a limit of 8 ticks (57 to 64 cycles, depending on the prescaler phase).
Port 2 fired with its record: first=2 at detector time 169. The snapshot 44ab reads busy=0100, stuck=0100 and the low byte of the bench's cycle count. Port 0, busy at the start, had answered after 30 cycles, so it is not in the snapshot: the record shows the state at the moment of the hang, not at the start of the wait.
After the functional reset the record is unchanged. The system logic was reset; the detector was not.
After a second hang, on port 1, its bit is set (0110) and the first-hang record still names port 2 at 169. Clearing and re-arming are checked at the end of the bench. Verilator 5.048 prints the same lines.
Each promise above is a check that can fail. Two plausible design mistakes, each in a scratch copy of the bench:
$ vvp -n mr.vvp # the detector reset tied to the functional reset: .dbg_rst_n(dbg_rst_n && rst_n)
after the functional reset: hung=0000 first=0 ts=0 snap=0000
FAIL: record lost in the functional reset
...
$ vvp -n ml.vvp # limit_i = 2 ticks (16 cycles), a guess instead of a measurement
cycle 20: rst_req: hung=1010 first=1 ts=17 snap=a013
FAIL: a 40-cycle answer was flagged as a hang
...The first is the most common way to lose a hang record: one reset tree for everything. The second fires on legal work, and its recovery request would reset a healthy chip every time a slow answer took more than about 16 cycles.
$ yosys -p "read_verilog -sv hang_det.sv; chparam -set N 4 hang_det; synth -top hang_det; stat"
...
398 cells
...
82 $_DFFE_PN0P_
40 $_DFF_PN0_
...398 generic cells and 122 flip-flops for four ports. Most of it is the record: the 32-bit free-running timestamp and its 32-bit copy, and the 16-bit snapshot. The timers themselves are 8 bits each. As an illustrative comparison, that is a few hundred flops next to a block that may have tens of thousands; a narrower timestamp or a smaller snapshot halves it.
Wired to dbglab's two master ports through hierarchical names, as a design would carry it, with a snapshot of the fabric, the DMA and the FIFO:
// not-run: needs dbglab's sources and bench, and hang_det.sv; compile it next to them (command in the Evidence box)
// det_probe: hang_det wired to dbglab's two master ports through hierarchical names, as the
// design would carry it. It changes nothing; it prints the detector's record when it fires.
`timescale 1ns/1ps
module det_probe;
logic [1:0] pend, prog, hung;
logic first, rst_req;
logic [31:0] first_ts;
logic [15:0] snap;
assign pend[0] = dbglab_tb.dut.m_valid[0]; // CPU request waiting
assign prog[0] = dbglab_tb.dut.m_ready[0] || dbglab_tb.dut.m_rvalid[0];
assign pend[1] = dbglab_tb.dut.u_dma.busy; // DMA run in progress
assign prog[1] = (dbglab_tb.dut.m_valid[1] && dbglab_tb.dut.m_ready[1]) || dbglab_tb.dut.m_rvalid[1];
hang_det #(.N(2), .PW(3), .TW(8), .SW(16)) u_det (.clk(dbglab_tb.clk), .dbg_rst_n(dbglab_tb.rst_n),
.en_i(1'b1), .clr_i(1'b0), .limit_i(8'd8), .pend_i(pend), .prog_i(prog),
.snap_i({dbglab_tb.dut.u_fab.grant, dbglab_tb.dut.m_lock, dbglab_tb.dut.u_dma.state,
dbglab_tb.dut.dreq, dbglab_tb.dut.u_per.wfull, dbglab_tb.dut.u_per.pctrl_q[0], 5'b0}),
.hung_o(hung), .first_o(first), .first_ts_o(first_ts), .snap_o(snap), .rst_req_o(rst_req));
always @(posedge dbglab_tb.clk) if (rst_req)
$display("HANGDET cycle=%0d: hung=%b first=%s at ts=%0d; snap: grant=%b lock=%b dma_state=%0d dreq=%b full=%b sink_en=%b",
dbglab_tb.cycle, hung, first ? "DMA" : "CPU", first_ts, snap[15:14], snap[13:12], snap[11:8],
snap[7], snap[6], snap[5]);
endmodule$ iverilog -g2012 -Wno-timescale -I tb -o build/det.vvp $(grep -v '^#' rtl/files.f) tb/bus_rules.sv tb/dbglab_tb.sv hang_det.sv det_probe.sv
$ vvp -n build/det.vvp +TEST=dma_burst_backpressure +SEED=1 | grep -E "^(dbglab|PASS|FAIL|HANGDET)"
dbglab: test=dma_burst_backpressure seed=1
HANGDET cycle=214: hung=01 first=CPU at ts=209; snap: grant=10 lock=10 dma_state=6 dreq=0 full=1 sink_en=0
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008Cycle 214, not 2165. The CPU's request has been pending since cycle 157 with no ready; the detector counts only that wait, so a limit of 8 ticks (it fired 57 cycles in) is safe where the bench needed 2,000 for all its idle stretches.
The snapshot is the wait-for table: the DMA owns the grant and holds lock, it is in state 6 (WAIT) with dreq=0, the FIFO is full and the sink is off. Everything the case study found with a probe, in one 16-bit register.
Silent elsewhere: on smoke, dma_copy, periph_stream and random seeds 1 to 3, and random seeds 4 to 20. (irq_w1c and periph_power, seed 1, fail with their own bugs' messages, not with HANGDET.)
On silicon, this is the register the field team reads after the next watchdog reset.
The bench showed each promise holding, and the mutations showed each one failing when the design is built the obvious wrong way. On dbglab the detector fired 1,951 cycles before the bench watchdog, at a limit about 30 times tighter, because it times waits rather than silence, and its snapshot carried the whole wait-for table. The cost is a few hundred cells, most of them in the record, which is the part that pays for itself.
It times only waits: a port that is pending without progress. The bench watchdog times silence on the whole bus, so it must allow for every legal idle stretch, such as the 302-cycle power-down in periph_power. A CPU that is idle, or a DMA that is between runs, never starts a detector timer.
Then the record must be read before that clear, by something that runs before software is back, which is rarely true in the field. Keep the record sticky until software has read it and clears it with an explicit write, and let later hangs set their own bits so a second event is visible without overwriting the first.
hang_det_tb.sv as shown, 2026-10-18: iverilog -g2012 -o h.vvp hang_det_tb.sv && vvp -n h.vvp (Icarus Verilog 13.0) and verilator --binary --timing --assert -Wno-fatal --top-module hang_det_tb (Verilator 5.048) print the lines shown and PASS. Mutations: the detector on dbg_rst_n && rst_n fails 2 checks; limit_i of 2 flags the legal 40-cycle answer and then times out at cycle 5001.
Area: yosys -p "read_verilog -sv hang_det.sv; chparam -set N 4 hang_det; synth -top hang_det; stat" (Yosys 0.69): 398 generic cells, 122 flip-flops (82 with enable, 40 without).
dbglab at HEAD 49c5f04: det_probe.sv and hang_det.sv compiled next to the bench (iverilog -g2012 -Wno-timescale -I tb -o build/det.vvp <rtl/files.f> tb/bus_rules.sv tb/dbglab_tb.sv hang_det.sv det_probe.sv), vvp -n build/det.vvp +TEST=dma_burst_backpressure +SEED=1: HANGDET at cycle 214, WATCHDOG at 2165. Silent on smoke, dma_copy, periph_stream and random seeds 1 to 3, and random seeds 4 to 20. Limits: Icarus only for the probe; generic cells, not a library.