A test that ran for an hour last week now ends after ten minutes with one line: WATCHDOG: nothing moved for 100 cycles. No assertion fired before it, no scoreboard mismatch, no X. The waveform after the last transfer is flat lines all the way to the end.
There is no first wrong value to walk back from. Where do you start?
Two moves are common, and both lose a day. Scrolling the flat waveform for the stuck signal finds nothing, because every signal is stuck. Raising the watchdog limit, "in case it was just slow", only makes a deadlock fail later with the same line.
In the log: silence, then one watchdog or timeout line. No error comes before it, because nothing is wrong; something is missing.
In counters: every work counter freezes at the same snapshot; only wait counters keep climbing, one per cycle (the counters lesson shows this on dbglab).
In a regression: the same test and cycle every time for a deadlock that needs one condition; a rate that grows with run length for a leak, because the leak needs several losses before it hangs.
If things still move but nothing finishes, it is not a hang: see starvation and livelock.
Snapshot every agent at the hang and write the wait-for table: who is waiting, for what, and the signal values that prove it. Read state registers, valids and readies, credit counters and FIFO levels, not the waveform's shape.
Follow the waits. If the chain comes back to where it started, it is a cyclic deadlock: find the arrow that is a design choice (holding a grant, a lock or a buffer while waiting) rather than a protocol rule. If the chain ends at an agent that is idle and holds nothing, a credit or token was lost: count the credits.
Find when the trap closed, not when the watchdog fired. For a cycle, the first cycle in which every edge was present; for a leak, the first cycle in which the credit count is wrong. That cycle, not the watchdog's, is the one to debug.
dbglab's B2 is the textbook cyclic case. At HEAD, dma_burst_backpressure hangs in both simulators:
$ tb/run.sh dma_burst_backpressure 1
dbglab: test=dma_burst_backpressure seed=1
FAIL test=dma_burst_backpressure seed=1 cycle=2165: WATCHDOG: no bus or stream progress for 2000 cycles; cpu_valid=1 cpu_ready=0 cpu_addr=00003008The wait-for table at the hang closes a loop of four: the CPU waits for the bus, the fabric keeps the bus for the DMA because it holds lock, the DMA waits for FIFO space, and the FIFO drains only after the CPU's write. Three arrows are normal behaviour; one, keep the grant while the owner waits, came from a performance commit. The whole hunt, with the probe, the bisect, a fix that looks right and still hangs, and the check that fires 1,984 cycles earlier, is the deadlock case study. Read it as the worked example of checks 1 to 3.
The other half of the class has no cycle at all. A sender may send only while it holds a credit; it starts with one per receiver buffer entry, spends one per word, and gets one back each time the receiver drains a word. The bug is a plausible power or timing tweak: the credit counter is only updated while the sender has data to send, so a credit that comes back during an idle gap is dropped.
// credit_tx: sends words over a link to a receiver with DEPTH buffer entries.
// It starts with DEPTH credits, spends one per word sent and gets one back per word drained.
// FIX=0 is the bug: the counter is only enabled while the sender has data (an
// activity-based enable), so a credit that returns while the sender is idle is lost.
module credit_tx #(parameter bit FIX = 1, parameter int DEPTH = 4) (
input logic clk, rst_n,
input logic have_data, // the sender has a word to send
output logic send, // a word goes on the link this cycle
input logic credit_ret, // one credit back from the receiver
output logic [2:0] credits
);
assign send = have_data && (credits != 3'd0);
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) credits <= 3'(DEPTH);
else if (FIX || have_data) credits <= credits - 3'(send) + 3'(credit_ret);
endmodule
// credit_rx: a DEPTH-entry buffer; drains a word when the consumer is ready and
// returns its credit one cycle later (a flop on the return path).
module credit_rx #(parameter int DEPTH = 4) (
input logic clk, rst_n,
input logic push, // a word arrives
input logic [7:0] din,
input logic take, // the consumer wants a word
output logic pop,
output logic [7:0] dout,
output logic credit_ret,
output logic [2:0] level
);
logic [7:0] mem [DEPTH];
logic [1:0] wp, rp;
assign pop = take && (level != 3'd0);
assign dout = mem[rp];
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) begin wp <= '0; rp <= '0; level <= '0; credit_ret <= 1'b0; end
else begin
if (push) begin mem[wp] <= din; wp <= wp + 2'd1; end
if (pop) rp <= rp + 2'd1;
level <= level + 3'(push) - 3'(pop);
credit_ret <= pop;
end
endmodule
First the fixed design, then the bug with only the watchdog, as most benches would have it:
$ iverilog -g2012 -o c.vvp credit_tb.sv && vvp -n c.vvp
PASS: 200 words in 391 cycles, credits back to 4 (FIX=1)
...
$ iverilog -g2012 -Pcredit_tb.FIX=0 -o cb.vvp credit_tb.sv
$ vvp -n cb.vvp +NOINV
FAIL cycle=143: WATCHDOG: nothing moved for 100 cycles; sent=16 got=16 credits=0 level=0
...Build the wait-for table from that one line. The sender has words left (sent=16 of 200) and waits for a credit: credits=0. Credits come back from drains, so it waits for the receiver. The receiver is empty (level=0, and got=16 equals sent), so it waits for a word. The chain is sender waits for a credit that nobody holds. It ends at an idle receiver, not back at the sender: no cycle, so check 2 says count the credits. Four entries, zero credits, zero words, nothing returning: four credits are missing.
Now the same run with the conservation check on:
$ vvp -n cb.vvp
FAIL cycle=11: CREDIT: credits=1 + level=2 + returning=0 != 4 (sent=8 got=6)
...Cycle 11, not 143. The check fires at the first lost credit, 132 cycles before the watchdog and 33 cycles before the last credit is gone.
`sent=8`: the first burst has just ended, so the sender's gap has begun. A credit came back in the gap and was dropped. That points straight at the counter's enable.
The sum is 3, not 4. A probe of the same run shows it at 1 by cycle 15: the second burst crawls through on a single credit, about one word every three cycles, and the last credit goes in the next gap at cycle 44. Here the slowdown lasts 30 cycles; in a real leak it can last hours and look like a performance bug.
Over seeds 1 to 10 the fixed design passes every time and the bug hangs every time, at cycles 140 to 165. Verilator 5.048 prints the same lines.
Cyclic deadlock: remove the arrow that is a choice. Never hold a shared resource (grant, lock, buffer slot) while waiting for an event that may need it; or give the waiting traffic its own resource (a separate queue, a reserved credit).
Lost credit or token: make the counter update on every cycle the event can happen, with no enable that depends on something else (credits <= credits - send + ret). A credit return is an event that must never be gated, merged or dropped.
For credits, the invariant in the bench above: credits + level + returning == DEPTH, as an assertion or a bench check on every credit loop. For locks and grants, the rule from the case study: no owner holds the grant idle while another master waits. Both catch the cause at its first cycle. Keep a watchdog too, sized from measurements (see timeouts and watchdogs), as the net under everything you did not think of. In review, ask of every credit, lock and grant: what does the holder wait for while it holds it, and can every return path be gated?
The chain ends at an idle slave that holds nothing, so it is not a cycle: a response (a token) was lost or never sent, or sent to the wrong master. Check S's request count against its responses and the fabric's routing of the response. A protocol checker counting requests and responses per port would have reported it when it happened.
It hides the leak instead of stopping it, and it can create credits that do not exist: if words are still in the buffer when the reset happens, the sender will overrun the receiver. The invariant still fires at cycle 11. The fix is to count every return, with no enable on the counter.
dbglab at HEAD 49c5f04 (built with dbglab/build-history.sh), 2026-10-18: tb/run.sh dma_burst_backpressure 1 and the same with verilator both print the WATCHDOG line at cycle 2165. The wait-for table and the bisect are the case study's (2026-10-17).
Credit reproduction (credit_tb.sv, on this page), Icarus Verilog 13.0: iverilog -g2012 -o c.vvp credit_tb.sv && vvp -n c.vvp prints PASS in 391 cycles. iverilog -g2012 -Pcredit_tb.FIX=0 -o cb.vvp credit_tb.sv, then vvp -n cb.vvp +NOINV (WATCHDOG at cycle 143) and vvp -n cb.vvp (CREDIT at cycle 11). Seeds 1 to 10: the fixed design passes all; the bug hangs all, at cycles 140 to 165. Verilator 5.048 (-GFIX=0 for the bug) prints the same lines for seeds 1 to 3. The credit sum over time comes from a copy of the bench with a $display of credits + level + credit_ret per cycle (4 until cycle 10, 1 from cycle 15, 0 from cycle 44).
Limits: one small link with a four-entry buffer, simulation only. A real credit leak can take hours; the invariant still fires at the first lost credit, which is the point.