The SoC boots its firmware on an FPGA prototype and runs a traffic test. Once in a few runs, after about forty minutes, the DMA stops issuing requests and never recovers. At the prototype's speed that is billions of cycles; in RTL simulation it would take years (illustrative orders of magnitude).
The prototype has a logic analyser built into the FPGA: a few dozen probe bits, a trace buffer a few thousand samples deep, and a trigger. Which signals do you probe, and what do you trigger on?
The obvious move is to probe the symptom and trigger on the hang: no request issued for a long time. The trigger fires, and the buffer holds a few thousand idle cycles; the bug happened earlier, outside the window. So the buffer is made deeper or probes are added, a rebuild of hours on an FPGA, and the window is still in the wrong place. The other habit is to dump everything, which an emulator allows for a window at a large cost in speed and an FPGA does not allow at all.
The real question is the one from bisection in time: when did it first go wrong? A trace buffer answers that only if you aim it.
Platform | Speed (illustrative) | What you can see | Changing what you see |
|---|---|---|---|
RTL simulation | hundreds of Hz to kHz for an SoC | every signal, every cycle | a recompile: seconds to minutes |
Emulator | around a MHz | most signals, over a captured window, at a cost in speed | often without a rebuild; capture depth is limited |
FPGA prototype | tens of MHz | only the probes chosen at build time, in an on-chip buffer | a rebuild: hours |
Silicon | full speed | pins, debug registers, scan dumps, on-chip trace if designed in | a new chip, or design-in for the next one |
Every row trades visibility for speed; the method is the same in each, and only the cost of an experiment changes. On an FPGA an experiment is a run with a trigger and a new probe is a rebuild, so you plan more carefully and lean on what fast platforms do well: running long, and running the same thing again.
Probe for the hypothesis, not only the symptom. Spend probe bits on the state your hypothesis says went wrong first (a counter, a state register, a handshake), even at the cost of some symptom signals.
Trigger on the symptom once, to learn when. The first run's job is a cycle number. It rarely shows the cause.
Then trigger on what must have happened earlier. Each window suggests what came before it: a counter wrapped, two events coincided. That is the next trigger. Sequence triggers (A, then B within N cycles) and occurrence counters (the 3rd A) make it precise.
Place the trigger in the window: mostly-before on a consequence, mostly-after on a suspected cause. (The model's window is fixed at 11 before and 4 after, so run 3 does not follow this; one sample after its cause was enough.)
Insist on deterministic replay. Walking back works only if each run reaches the same cycle in the same state. Control or record every external input (recording everything to replay) and check that one trigger fires on the same cycle twice.
Bring it back to simulation as soon as a window shows the first wrong cycle: a directed test that reaches that state in a few hundred cycles (moving the failure to the cheapest stage).
On the debug loop this is Observe and Localise with a very small window: each trigger is a bisection in time step that you choose instead of computing.
The run is 83,047 cycles long and each window is 16 samples, so one window sees less than 0.02 % of the run. The three windows do not overlap: each run had to be aimed by what the previous one showed.
outs_ctr counts requests still waiting for a response and stops issuing at 4. Its bug: when a request and a response come on the same cycle, it counts only the response. That loses one request; later a response arrives while the counter reads 0, the counter wraps to 7, and the block never issues again. Requests and responses are rare, so the coincidence is rare. trace_buf is the trace-buffer model. The bench runs the same seeded traffic four times, as four replays would: three buggy runs with different triggers, and one fixed run.
`timescale 1ns/1ps
// outs_ctr: counts requests still waiting for a response, and stops issuing at 4.
// The bug: a request and a response on the same cycle count as a response only, so the
// counter loses a request. Later a response arrives with outs = 0, outs wraps to 7, and the
// block never issues again.
module outs_ctr #(parameter bit FIX = 0) (
input logic clk, rst_n,
input logic want, // a new request is wanted
input logic rsp, // a response returns
output logic req, // a request is issued
output logic [2:0] outs
);
assign req = want && (outs < 3'd4);
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) outs <= '0;
else if (FIX && req && rsp) outs <= outs; // the fix: both together leave it alone
else if (rsp) outs <= outs - 3'd1; // bad (FIX = 0): req && rsp counts as rsp only
else if (req) outs <= outs + 3'd1;
endmodule
// trace_buf: a model of an emulator's or FPGA's on-chip trace buffer: NPROBE bits sampled every
// cycle into a ring of DEPTH entries. When the trigger fires it keeps POST more samples, then
// freezes; the window is read out afterwards, as over JTAG. Nothing else is visible.
module trace_buf #(parameter int NPROBE = 8, DEPTH = 16, POST = 4) (
input logic clk, rst_n,
input logic [NPROBE-1:0] probe,
input logic trig,
output logic frozen
);
logic [NPROBE-1:0] ring [DEPTH];
int unsigned cyc [DEPTH]; // the cycle number of each sample (a timestamp counter)
int unsigned wp, left, now, trig_at;
logic armed;
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) begin
wp <= 0; now <= 0; armed <= 1'b1; frozen <= 1'b0; left <= 0; trig_at <= 0;
end else begin
now <= now + 1;
if (!frozen) begin
ring[wp] <= probe; cyc[wp] <= now;
wp <= (wp + 1) % DEPTH;
if (armed && trig) begin armed <= 1'b0; left <= POST; trig_at <= now; end
else if (!armed) begin
if (left == 1) frozen <= 1'b1;
left <= left - 1;
end
end
end
// Read the window out, oldest sample first. The probe word is {req, rsp, want, outs[2:0], 2'b0}.
task automatic readout(input string name);
$display("%s: trigger at cycle %0d, %0d samples before it and %0d after", name, trig_at, DEPTH - POST - 1, POST);
for (int k = 0; k < DEPTH; k++) begin
int unsigned i = (wp + k) % DEPTH;
$display(" cycle %7d req=%b rsp=%b want=%b outs=%0d%s", cyc[i], ring[i][7], ring[i][6], ring[i][5],
ring[i][4:2], cyc[i] == trig_at ? " <- trigger" : "");
end
endtask
endmodule
// tb: three runs of the same seeded traffic, as three emulation runs replay the same stimulus,
// each with a different trigger. Run 1: the symptom (16 wanted requests refused since the last issue).
// Run 2: the counter wraps (outs == 7). Run 3: the first cycle with req and rsp together.
// A fourth copy has the fix and the symptom trigger; it must never fire in 200,000 cycles.
module tb;
logic clk = 1'b0, rst_n = 1'b0;
always #5 clk = ~clk;
for (genvar r = 0; r < 4; r++) begin : g_run
logic want, rsp, req, trig, frozen, rsp_q;
logic [2:0] outs;
int unsigned rng = 32'h1234_5678, real_outs = 0, stall = 0;
outs_ctr #(.FIX(r == 3)) u_dut (.clk, .rst_n, .want, .rsp, .req, .outs);
trace_buf #(.NPROBE(8), .DEPTH(16), .POST(4)) u_tb (.clk, .rst_n, .probe({req, rsp, want, outs, 2'b00}), .trig, .frozen);
// The world around it: a request is wanted in 1 cycle of 256; a response returns for a real
// outstanding request in 1 cycle of 128. Every run uses the same seed, so the same traffic.
always_ff @(posedge clk) begin
rng = rng ^ (rng << 13); rng = rng ^ (rng >> 17); rng = rng ^ (rng << 5);
want <= (rng[7:0] == 8'd0);
rsp_q <= (rng[14:8] == 7'd0);
end
assign rsp = rsp_q && (real_outs > 0);
always_ff @(posedge clk) if (rst_n) begin
real_outs <= real_outs + (req ? 1 : 0) - (rsp ? 1 : 0);
stall <= (want && !req) ? stall + 1 : (req ? 0 : stall);
end
assign trig = (r == 0 || r == 3) ? (stall >= 16) : (r == 1) ? (outs == 3'd7) : (req && rsp);
end
initial begin
repeat (2) @(posedge clk);
#1 rst_n = 1'b1;
wait (g_run[0].u_tb.frozen && g_run[1].u_tb.frozen && g_run[2].u_tb.frozen);
g_run[0].u_tb.readout("run 1, trigger on the symptom");
g_run[1].u_tb.readout("run 2, trigger on outs == 7");
g_run[2].u_tb.readout("run 3, trigger on req && rsp");
while (g_run[3].u_tb.now < 200000) @(posedge clk);
$display("fix, trigger on the symptom: %s after %0d cycles", g_run[3].u_tb.frozen ? "FIRED" : "never fired", g_run[3].u_tb.now);
if (g_run[3].u_tb.frozen) $fatal(1, "the fixed counter stalled");
if (!(g_run[2].u_tb.trig_at + 16 < g_run[1].u_tb.trig_at && g_run[1].u_tb.trig_at + 16 < g_run[0].u_tb.trig_at))
$fatal(1, "the lesson's claims do not hold");
$display("PASS: each 16-sample window misses the step before it; three replays walk back from the hang to its cause");
$finish;
end
endmodule$ iverilog -g2012 -o t.vvp trace.sv && vvp -n t.vvprun 1, trigger on the symptom: trigger at cycle 83043, 11 samples before it and 4 after
cycle 83032 req=0 rsp=0 want=0 outs=7
cycle 83033 req=0 rsp=0 want=0 outs=7
...
cycle 83042 req=0 rsp=0 want=1 outs=7
cycle 83043 req=0 rsp=0 want=0 outs=7 <- trigger
...
cycle 83047 req=0 rsp=0 want=0 outs=7The trigger fired when 16 wanted requests had been refused since the last issue: want rises at 83,042, req stays 0. outs reads 7 in every sample, which is impossible in a block that stops at 4. So this run gives a time (the trouble began before 83,032) and a clue (outs = 7), and nothing about how 7 happened.
Hypothesis: the counter went wrong at some moment and has been stuck since. Prediction: at that moment it changes to 7. Trigger on outs == 7.
run 2, trigger on outs == 7: trigger at cycle 79533, 11 samples before it and 4 after
cycle 79522 req=0 rsp=0 want=0 outs=0
...
cycle 79532 req=0 rsp=1 want=0 outs=0
cycle 79533 req=0 rsp=0 want=0 outs=7 <- trigger
...The same traffic, replayed, and the window has moved back 3,510 cycles. At 79,532 a response arrives while outs is 0, and the counter wraps to 7. A response needs a request, so the counter must have lost one earlier. Its obvious corner is a request and a response on the same cycle.
run 3, trigger on req && rsp: trigger at cycle 78644, 11 samples before it and 4 after
cycle 78633 req=0 rsp=0 want=0 outs=1
...
cycle 78643 req=0 rsp=0 want=0 outs=1
cycle 78644 req=1 rsp=1 want=1 outs=1 <- trigger
cycle 78645 req=0 rsp=0 want=0 outs=0
...The first coincidence in the run is at cycle 78,644, 889 cycles before the wrap. With one request outstanding, a new request and a response arrive together; the count should stay at 1 and it drops to 0. That is the first wrong cycle, and the RTL behind it is three lines. Notice that none of the windows could have shown the one before it: the cause is 889 cycles before the wrap and the wrap is 3,510 before the symptom, both far more than 16 samples.
fix, trigger on the symptom: never fired after 200000 cycles
PASS: each 16-sample window misses the step before it; three replays walk back from the hang to its causeWith FIX = 1 (both together leave the count alone) the symptom trigger never fires in 200,000 cycles of the same traffic, more than twice the length of the failing run. Verilator 5.048 prints the same output for all four runs. On a real platform the last step would be a directed simulation test: one outstanding request, then a request and a response on the same cycle, which reproduces the bug in ten cycles instead of 78,644.
A 16-sample window on an 83,000-cycle run found a one-cycle cause, because each run was aimed by a hypothesis and each replay reached the same cycles. Determinism did as much work as the trigger: without it, the windows of different runs could not be lined up.
outs = 0. Name two hypotheses that explain it, and the trigger that would split them.Either the counter lost a request earlier (it undercounted), or a spurious response arrived for a request that never existed (the responder misbehaved). They predict different things: an undercount means some earlier cycle where outs failed to go up for an issued request; a spurious response means the number of responses exceeds the number of requests. A trigger on "request and response together" (where an undercount is likely) or a sequence trigger on "response while the responder had nothing pending" tells them apart; here the first found the coincidence.
The run is not deterministic, so windows from different runs cannot be lined up, and a trigger aimed by one run may miss in the next. Find the source: an asynchronous clock, a timer, an external link, or uninitialised state. Control it or record it so it can be replayed, and confirm the trigger fires on the same cycle twice, before you use replays to walk back.
No emulator or FPGA is available or used. trace.sv as shown is a simulation model of a trace buffer (8 probe bits, 16 samples, 4 after the trigger) on a small block, 2026-10-18: iverilog -g2012 -o t.vvp trace.sv && vvp -n t.vvp (Icarus Verilog 13.0, 1.6 s) and verilator --binary --timing --assert -Wno-fatal --top-module tb trace.sv (Verilator 5.048): identical output. Triggers fire at cycles 83,043 (symptom), 79,533 (outs == 7) and 78,644 (req and rsp together); the fixed copy's symptom trigger never fires in 200,000 cycles; PASS.
Speeds, capacities and build times in the comparison table are illustrative orders of magnitude, not measurements; they vary widely with the design, the platform and the vendor.
Limits: the model has none of a real platform's costs (rebuild time, probe routing, clock limits, I/O speed bridges), and its stimulus is deterministic by construction. Real trace buffers are deeper and wider; the 16 samples are chosen so the output fits the page, and the walk-back works the same way at any depth.