Eight weeks to tape-out. The review of a request engine reaches its last item, debuggability, and the block owner reads a list: status register, hang timer, some counters. The bring-up lead asks the question that matters: "When it hangs on the bench and I read those registers, how do I tell whether your engine is stuck or the thing it talks to never answered?" Nobody in the room knows.
The features exist. What is missing is the part that makes them useful: which value means which culprit, and a test that shows the rule is right. How do you design that in, and how does a reviewer check it?
Debug features usually arrive as a generic set, copied from the last chip, added after the function is done. Each is plausible on its own. Together they fail in four ways. They report without deciding: a HANG bit says something stopped, not who stopped it. They share the bug: a status field that shows the engine's own outstanding count shows the wrong value with full confidence when that count is the bug. They are untested: nothing in the regression reads them, so a broken clear or a detector that fires on a slow partner ships unnoticed. And they are cut first, because nobody can say what is lost.
The features and their shapes (sticky errors, counters, trace rings, state readback, side-band access) are covered in observability by design, and the hang detector in watchdogs and hang detectors. This lesson is the step before and after them: deciding what they must answer, and proving that they do.
List the failure modes of the block from its interfaces and the bug classes: a partner that never answers, lost or duplicated credits, a stray or unexpected response, a stuck state, an overflow, a slow but healthy partner that must not be called a failure.
Write the debug spec. For each mode: the register fields the lab reads and the decision rule that turns them into a sentence ("if unanswered equals outstanding, the partner is silent"). If no rule can tell two modes apart, add a field.
Use independent witnesses. A diagnosis compares two quantities that one bug cannot corrupt together: requests issued and responses received, counted separately, against the engine's own outstanding count.
Keep the first event and read it without disturbing. A snapshot frozen at the first hang; sticky errors cleared only by an explicit write; reads with no side effects, through a path that works when the functional port is stuck.
Prove each rule with a diagnosis test. Inject the failure mode in simulation and check the decoded registers name it. Then mutate the debug logic: a mutant that passes is a promise nobody tested.
Budget it and sign it off with evidence, not with a list: the spec, the tests and the area line, reviewed like any other requirement.
On the debug loop this is Prevent, done before there is a bug, and it pays at Observe and Localise on silicon, where a register read replaces the probe you cannot add.
The engine issues len requests, keeps at most two outstanding and finishes when all are answered. Its debug spec is the figure above. A side-band port reads four registers: STATUS (hang, error, state, outstanding), COUNT (issued, answered, left), HANG (a snapshot of those taken at the first hang) and QUIET (cycles without progress). FIX = 0 is a real bug of the kind this design invites: when a request and a response land in the same cycle, the response's decrement is lost. The bench drives four tests and judges each one only through register reads, as the lab would.
// xfer_engine: issues LEN requests, at most two outstanding, and finishes when every request has
// been answered. Its debug features came from a review of how it can fail:
// - state, outstanding count and per-direction counters, readable at any time;
// - a hang detector: LIMIT cycles busy with no request and no response sets a sticky HANG bit
// and freezes a snapshot of the state at that moment (the first hang is kept);
// - a sticky error for a response that arrives when nothing is outstanding;
// - a side-band read port (dbg_addr / dbg_rdata) that works when the functional port is stuck.
// FIX = 0 is the bug: a request and a response in the same cycle lose the response's decrement.
`timescale 1ns/1ps
module xfer_engine #(parameter int LIMIT = 32, parameter bit FIX = 1) (
input logic clk, rst_n,
input logic start,
input logic [7:0] len,
output logic req_valid,
input logic req_ready,
input logic rsp_valid,
output logic busy, done,
input logic [1:0] dbg_addr,
output logic [31:0] dbg_rdata,
input logic dbg_clr // write-1-to-clear of HANG and ERR, one pulse
);
typedef enum logic [1:0] {IDLE = 2'd0, RUN = 2'd1, DRAIN = 2'd2} st_t;
st_t st;
logic [7:0] left, issued, completed;
logic [1:0] outst;
logic [15:0] quiet; // cycles busy without progress
logic hang, err;
logic [31:0] snap;
logic fire, prog, rsp_ok;
assign req_valid = (st == RUN) && (left != 0) && (outst < 2);
assign fire = req_valid && req_ready;
assign rsp_ok = rsp_valid && (outst != 0);
assign prog = fire || rsp_valid;
assign busy = (st != IDLE);
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) begin
st <= IDLE; left <= '0; outst <= '0; issued <= '0; completed <= '0; done <= 1'b0;
quiet <= '0; hang <= 1'b0; err <= 1'b0; snap <= '0;
end else begin
done <= 1'b0;
if (fire) begin left <= left - 1'b1; issued <= issued + 1'b1; end
if (rsp_valid) completed <= completed + 1'b1;
if (FIX) outst <= outst + 2'(fire) - 2'(rsp_ok);
else if (fire) outst <= outst + 1'b1; // bad when FIX = 0: a same-cycle response is lost
else if (rsp_ok) outst <= outst - 1'b1;
if (rsp_valid && outst == 0) err <= 1'b1; // a response nobody asked for: flag it, do not count it
case (st)
IDLE: if (start) begin st <= RUN; left <= len; issued <= '0; completed <= '0; end
RUN: if (left == 1 && fire || left == 0) st <= DRAIN;
DRAIN: if (outst == 0) begin st <= IDLE; done <= 1'b1; end
default: st <= IDLE;
endcase
// hang detector with a first-hang snapshot
quiet <= (st == IDLE || prog) ? 16'd0 : quiet + 16'd1;
if (quiet == 16'(LIMIT) && !hang) begin
hang <= 1'b1;
snap <= {2'b0, st, 2'b0, outst, issued, completed, left};
end
if (dbg_clr) begin hang <= 1'b0; err <= 1'b0; end
end
always_comb
case (dbg_addr)
2'd0: dbg_rdata = {hang, err, 24'b0, st, 2'b0, outst}; // STATUS
2'd1: dbg_rdata = {8'b0, issued, completed, left}; // COUNT
2'd2: dbg_rdata = snap; // HANG snapshot
default: dbg_rdata = {16'b0, quiet}; // QUIET
endcase
endmodule
// tb: four tests, judged only through side-band register reads, as the lab would.
// 1. the responder goes silent on purpose: the registers must blame the responder;
// 2. one response nobody asked for: the sticky ERR bit must set, and W1C must clear it;
// 3. a slow responder that keeps answering: it must not be called a hang;
// 4. 200 requests with random latency: no hang, and every count must balance.
`timescale 1ns/1ps
module tb;
parameter bit FIX = 1;
logic clk = 1'b0, rst_n = 1'b0, start = 1'b0, req_valid, rsp_valid = 1'b0, busy, done, dbg_clr = 1'b0;
logic stall = 1'b0, spurious = 1'b0, slow = 1'b0;
logic [7:0] len = '0;
logic [1:0] dbg_addr = '0;
logic [31:0] dbg_rdata;
int unsigned cyc = 0, due[$];
int errors = 0;
always #5 clk = ~clk;
always @(posedge clk) cyc <= cyc + 1;
xfer_engine #(.LIMIT(32), .FIX(FIX)) dut (.clk, .rst_n, .start, .len, .req_valid, .req_ready(1'b1),
.rsp_valid, .busy, .done, .dbg_addr, .dbg_rdata, .dbg_clr);
// responder: always ready; answers in order, 1 to 3 cycles after each request (or one answer
// per 20 cycles while slow is high); silent while stall is high
always @(posedge clk) begin
if (req_valid) due.push_back(slow ? (due.size() != 0 ? due[$] : cyc) + 20 // test 3: one answer per 20 cycles
: cyc + 1 + $urandom_range(0, 2));
rsp_valid <= spurious; // test 2: one response nobody asked for
if (!stall && due.size() != 0 && due[0] <= cyc) begin void'(due.pop_front()); rsp_valid <= 1'b1; end
end
task automatic rd(input logic [1:0] a, output logic [31:0] v); // one side-band read
dbg_addr = a; #1 v = dbg_rdata;
endtask
// the lab's view: decode the registers into a sentence
function automatic string sname(input logic [1:0] v);
return v == 2'd0 ? "IDLE" : v == 2'd1 ? "RUN" : v == 2'd2 ? "DRAIN" : "?";
endfunction
task automatic diagnose(output int kind, output string why);
logic [31:0] s, h;
logic [7:0] unanswered;
rd(2'd0, s); rd(2'd2, h);
unanswered = h[23:16] - h[15:8];
if (!s[31]) begin kind = 0; why = "healthy: no hang recorded"; end
else if (unanswered == 8'(h[25:24]) && unanswered != 0) begin
kind = 1;
why = $sformatf("waiting on the responder: %0d request(s) issued and not answered, hung in %s", unanswered, sname(h[29:28]));
end else begin
kind = 2;
why = $sformatf("accounting error in the engine: issued %0d, answered %0d, yet outstanding = %0d in %s", h[23:16], h[15:8], h[25:24], sname(h[29:28]));
end
endtask
task automatic show_regs();
logic [31:0] s, c, h, q;
rd(2'd0, s); rd(2'd1, c); rd(2'd2, h); rd(2'd3, q);
$display(" STATUS=%h COUNT=%h HANG=%h QUIET=%0d", s, c, h, q);
endtask
task automatic wait_idle_or_hang(output logic [31:0] v); // until done, a recorded hang, or 5000 cycles
for (int t = 0; t < 5000; t++) begin
@(negedge clk);
rd(2'd0, v);
if (!busy || v[31]) break;
end
@(negedge clk);
endtask
task automatic run(input logic [7:0] n);
@(negedge clk) len = n; start = 1'b1; @(negedge clk) start = 1'b0;
endtask
initial begin
int kind;
string d;
logic [31:0] v;
repeat (2) @(negedge clk); rst_n = 1'b1;
// test 1: a responder that stops on purpose; the debug logic must blame the responder
stall = 1'b1;
run(8'd2);
repeat (60) @(negedge clk);
diagnose(kind, d);
$display("test 1 (responder silent for 60 cycles): %s", d);
show_regs();
if (kind != 1) begin $display("FAIL: test 1 should blame the responder"); errors++; end
@(negedge clk) stall = 1'b0; dbg_clr = 1'b1;
@(negedge clk) dbg_clr = 1'b0;
wait_idle_or_hang(v);
rd(2'd1, v);
if (busy || v !== {8'b0, 8'd2, 8'd2, 8'd0}) begin
$display("FAIL: test 1 did not finish clean"); show_regs(); errors++;
end
// test 2: a response while nothing is outstanding must set the sticky ERR bit, and W1C clears it
@(negedge clk) spurious = 1'b1;
@(negedge clk) spurious = 1'b0;
repeat (3) @(negedge clk);
rd(2'd0, v);
$display("test 2 (one response with nothing outstanding): STATUS=%h", v);
if (v !== 32'h4000_0000) begin $display("FAIL: test 2 should set ERR only"); errors++; end
dbg_clr = 1'b1; @(negedge clk) dbg_clr = 1'b0;
rd(2'd0, v);
if (v !== 32'h0) begin $display("FAIL: W1C did not clear ERR"); errors++; end
// test 3: a slow responder that keeps answering (one per 20 cycles) must not be called a hang
slow = 1'b1;
run(8'd4);
wait_idle_or_hang(v);
slow = 1'b0;
diagnose(kind, d);
$display("test 3 (slow but alive responder) at cycle %0d: %s", cyc, d);
rd(2'd1, v);
if (busy || kind != 0 || v !== {8'b0, 8'd4, 8'd4, 8'd0}) begin $display("FAIL: test 3"); show_regs(); errors++; end
// test 4: 200 requests with random latency
run(8'd200);
wait_idle_or_hang(v);
diagnose(kind, d);
$display("test 4 (200 requests, FIX=%0d) at cycle %0d: %s", FIX, cyc, d);
show_regs();
if (busy || kind != 0) errors++;
rd(2'd1, v);
if (v !== {8'b0, 8'd200, 8'd200, 8'd0}) errors++;
if (errors != 0) $fatal(1, "%0d check(s) failed", errors);
$display("PASS: the registers blamed the silent responder, flagged the stray response, spared the slow one, and 200 requests finished clean");
$finish;
end
endmodule$ iverilog -g2012 -o x1.vvp xfer_dbg.sv && vvp -n x1.vvp
test 1 (responder silent for 60 cycles): waiting on the responder: 2 request(s) issued and not answered, hung in DRAIN
STATUS=80000022 COUNT=00020000 HANG=22020000 QUIET=59
test 2 (one response with nothing outstanding): STATUS=40000000
test 3 (slow but alive responder) at cycle 162: healthy: no hang recorded
test 4 (200 requests, FIX=1) at cycle 588: healthy: no hang recorded
STATUS=00000000 COUNT=00c8c800 HANG=22020000 QUIET=0
PASS: the registers blamed the silent responder, flagged the stray response, spared the slow one, and 200 requests finished clean
xfer_dbg.sv:195: $finish called at 5887000 (1ps)Read test 1's registers as hex fields. HANG=22020000: state 2 (DRAIN), outstanding 2, issued 0x02, answered 0x00. Two requests out, none back, and the engine's own count agrees: the rule says the responder is silent, which is what the bench did. Test 2 sets ERR and nothing else (40000000). Test 3 is the false alarm a careless detector raises: answers every 20 cycles, a long wait for a 32-cycle limit, and no hang, because progress on either side resets the timer. Test 4 balances: 0xc8 issued, 0xc8 answered, nothing left. Verilator prints the same lines (test 4 ends at cycle 596, from its own random latencies) and PASS.
$ iverilog -g2012 -P tb.FIX=0 -o x0.vvp xfer_dbg.sv && vvp -n x0.vvp
...
test 3 (slow but alive responder) at cycle 162: healthy: no hang recorded
test 4 (200 requests, FIX=0) at cycle 1001: accounting error in the engine: issued 200, answered 200, yet outstanding = 1 in DRAIN
STATUS=80000021 COUNT=00c8c800 HANG=21c8c800 QUIET=35
FATAL: xfer_dbg.sv:193: 1 check(s) failed
Time: 10017000 Scope: tb.$unm_blk_14This is the read the bring-up lead asked for. HANG=21c8c800: the engine is in DRAIN, waiting for 1 outstanding response, yet it issued 0xc8 requests and received 0xc8 responses. The responder has answered everything; the engine's own count is wrong, and only independent witnesses (issued and completed, which the bug cannot reach) could show it. One read moves the search from "the system hung" to the outstanding counter's update, and in this design only one line can lose a decrement. Notice also cycle 1001 against 588 in the healthy run: as the count drifted up, the engine throttled itself to one request at a time. A throughput counter would have flagged this chip as slow before it ever hung.
A promise in the debug spec that no test checks is a guess. Mutate the debug logic and see which tests notice:
# mutant 1: W1C no longer clears HANG
sed "s/if (dbg_clr) begin hang <= 1'b0; err <= 1'b0; end/if (dbg_clr) begin err <= 1'b0; end/" xfer_dbg.sv > mut_clr.sv
# mutant 2: the hang detector counts only requests as progress
sed 's/ assign prog = fire || rsp_valid;/ assign prog = fire;/' xfer_dbg.sv > mut_prog.sv$ iverilog -g2012 -o m1.vvp mut_clr.sv && vvp -n m1.vvp
...
FATAL: mut_clr.sv:193: 6 check(s) failed
...
$ iverilog -g2012 -o m2.vvp mut_prog.sv && vvp -n m2.vvp
...
test 3 (slow but alive responder) at cycle 155: waiting on the responder: 1 request(s) issued and not answered, hung in DRAIN
FAIL: test 3
...
FATAL: mut_prog.sv:193: 3 check(s) failed
...Both mutants fail now. The second one has a history worth telling: the first version of this bench had no test 3, and the mutant passed. A detector that ignored responses would have shipped, and on silicon it would have called every slow but healthy partner a hang, which teaches the lab to ignore the HANG bit. The surviving mutant named the missing test.
$ yosys -q -p "read_verilog -sv eng.sv; synth -top xfer_engine; tee -o eng.stat stat"
$ yosys -q -p "read_verilog -sv eng_nodbg.sv; synth -top xfer_engine; tee -o eng_nodbg.stat stat"
$ grep -E "cells$|DFF" eng.stat eng_nodbg.stat
eng.stat: 390 cells
eng.stat: 32 $_DFFE_PN0P_
eng.stat: 43 $_DFF_PN0_
eng_nodbg.stat: 78 cells
eng_nodbg.stat: 2 $_DFFE_PN0P_
eng_nodbg.stat: 11 $_DFF_PN0_eng_nodbg.sv ties dbg_rdata to zero, so synthesis removes everything only the debug port reads: the function is 13 flip-flops and the debug features are the other 62 (two 8-bit counters, a 16-bit quiet timer, 28 live snapshot bits and two sticky bits). In this toy that is five times the function; in a real engine the same 62 flops are a few percent, and they scale with the number of interfaces, so budget per interface.
Use it at the design review, block by block. The middle column is what makes it a sign-off rather than a conversation: evidence that can be opened and run.
Item | Ask | Evidence to show |
|---|---|---|
Status and debug registers | Which register shows each failure mode, and what rule decodes it? | the debug spec: mode, fields, decision rule |
Counters | Which quantities are counted independently on each side of an interface? | issued and completed, or sent and credited, as separate fields |
Event trace buffers | What triggers and freezes the trace, and how deep is it? | trigger list, depth chosen from a failure's history length |
Watchdogs and hang detectors | What counts as progress, and what is the limit's slowest legal partner? | a slow-partner test that does not fire |
Observable state machines | Can every FSM state and the condition it waits for be read? | state and wait-reason fields, read-only |
Sticky error bits | Is the first error kept, with when and why, and cleared only by software? | first-error record; W1C test |
Read path | Can all of it be read when the functional bus or clock is stuck? | side-band or JTAG path, on an always-on clock |
Diagnosis tests | Does a test inject each mode and check the decoded answer? | one test per spec row; mutation results |
Budget and access | What does it cost, and who can read it in production? | area line per feature; access control agreed with security |
Each injected failure is named from register reads alone, and a slow partner is not a failure. One read separates "the partner never answered" from "the engine lost count", a distinction that on silicon could take days of bench time, and the mutations show the debug logic itself is tested.
issued and completed and keeping only outstanding in the HANG snapshot. Which test would still pass, which diagnosis is lost, and how would you argue it?Test 1 still looks right (outstanding 2, waiting), but the rule can no longer tell a silent responder from an engine that lost count: both show "waiting for n responses". The FIX = 0 run would be blamed on the responder. The argument is the bench time saved on the first such hang, against about 32 flops (the two live counters and their copies in the snapshot).
With progress counted on both sides, a 50-cycle gap with nothing happening still exceeds 32, so HANG sets on healthy traffic and the lab learns to ignore it. Make the limit programmable with a reset value above the slowest legal latency, document it in the spec, and add a bench case at that latency that must not fire, the way test 3 does for 20 cycles.
xfer_dbg.sv as shown (engine and bench in one file), 2026-10-18: iverilog -g2012 -o x1.vvp xfer_dbg.sv && vvp -n x1.vvp (Icarus Verilog 13.0) prints the four test lines and PASS; with -P tb.FIX=0 test 4 prints the accounting-error diagnosis and $fatals. verilator --binary --timing --assert -Wno-fatal --top-module tb (Verilator 5.048, no warnings) gives PASS, and with -GFIX=0 the same diagnosis at cycle 986 (Icarus 1001; the simulators draw different random latencies).
Mutations by sed, run in Icarus: W1C no longer clearing HANG fails 6 checks; a hang detector that counts only requests as progress fails 3 checks, and passed the first version of the bench, which had no slow-responder test. Area: yosys -q -p "read_verilog -sv eng.sv; synth -top xfer_engine; stat" (Yosys 0.69) on the engine alone: 390 generic cells, 75 flip-flops; with dbg_rdata tied to 0 so that the debug logic is removed: 78 cells, 13 flip-flops.
Limits: generic Yosys cells, not a library; the engine is tiny, so the debug logic's share of it is far larger than in a real block. The review meeting, its people and the area percentages in the lens are illustrative. No silicon or lab access is run here; see the bring-up course.