The FIFO writer from the previous lesson drops a word about once in a hundred runs at its nightly setting. Suppose this were an emulator run or a board: no reruns, no seeds, and a full waveform of an hour is out of the question. Even in this tiny simulation, dumping everything for 200 seeds writes 55 MB.
When it next happens, you want the cycles that led up to it, not just the scoreboard's complaint. What do you have to set up before it happens?
Three habits: dump everything and hope the disk holds; wait for the failure, then rerun the seed with a dump; or add a lot of logging and wait again. The first does not scale past small designs. The second only works when the run is deterministic and cheap, which is exactly what lab and emulation runs are not, and it fails for inputs you do not control, such as start values. The third changes the timing of the bench if it is done carelessly (heisenbugs) and still produces megabytes of text around one interesting line.
Run a flight recorder. Keep the last N cycles (or transactions) of the few signals your question needs in a ring buffer. It costs a write per cycle and nothing else until it is read.
Freeze it from the check that fails. The scoreboard or assertion that sees the symptom stops the recorder and prints it, so the history before the failure survives.
Size it to the detection latency. The ring must reach back from the moment the check fires to the cause. Measure that time on one capture and give it a margin of several times.
Choose the trigger. The symptom is always available and late. A precursor is a cheap, earlier sign that fires more often than the bug: count its false triggers. A cause trigger, an assertion on the exact fault, fires at the right cycle but needs a hypothesis first.
Dump windows, not runs. Keep the waveform off and switch it on for a fixed window when the trigger fires.
Check the capture changed nothing. The failing seeds and cycles must be the same with and without the recorder.
Trigger | Fires | Gives you | Costs you |
|---|---|---|---|
Symptom (the failing check) | Once, late | Proof of failure; the recorder's history | The cause may be older than the ring |
Precursor (here: the FIFO goes full) | Before the failure, and also when nothing fails | A window that starts at the cause | False triggers to sort and store |
Cause (here: a write while full) | At the faulty cycle only | The exact cycle, every time | You must already suspect it |
The bench below runs the nightly setting (seeds 1 to 200, the same two failures as before). Two recorders, 4 and 32 cycles deep, watch the writer's pins. The consumer's check is the symptom trigger: it freezes both recorders and tests whether each still holds the dropped write. The precursor is the FIFO going full, counted on every run, and with +DUMP=2 it opens a 40-cycle dump window.
// afifo: a compact version of dbglab's async FIFO (two-flop Gray pointer synchronisers, inlined), here 8 words deep
module afifo #(parameter int DW = 16, parameter int AW = 3) (
input logic wclk, wrst_n, winc,
input logic [DW-1:0] wdata,
output logic wfull,
input logic rclk, rrst_n, rinc,
output logic [DW-1:0] rdata,
output logic rempty
);
localparam logic [AW:0] TOP2 = 3 << (AW - 1);
logic [DW-1:0] mem [1 << AW];
logic [AW:0] wbin, wbin_n, wgray, wgray_n, rq1, rptr_w;
logic [AW:0] rbin, rbin_n, rgray, rgray_n, wq1, wptr_r;
assign wbin_n = wbin + (winc & ~wfull);
assign wgray_n = (wbin_n >> 1) ^ wbin_n;
always_ff @(posedge wclk or negedge wrst_n)
if (!wrst_n) begin wbin <= '0; wgray <= '0; wfull <= 1'b0; rq1 <= '0; rptr_w <= '0; end
else begin
wbin <= wbin_n; wgray <= wgray_n; wfull <= (wgray_n == (rptr_w ^ TOP2));
rq1 <= rgray; rptr_w <= rq1;
end
always_ff @(posedge wclk) if (winc && !wfull) mem[wbin[AW-1:0]] <= wdata;
assign rbin_n = rbin + (rinc & ~rempty);
assign rgray_n = (rbin_n >> 1) ^ rbin_n;
always_ff @(posedge rclk or negedge rrst_n)
if (!rrst_n) begin rbin <= '0; rgray <= '0; rempty <= 1'b1; wq1 <= '0; wptr_r <= '0; end
else begin
rbin <= rbin_n; rgray <= rgray_n; rempty <= (rgray_n == wptr_r);
wq1 <= wgray; wptr_r <= wq1;
end
assign rdata = mem[rbin[AW-1:0]];
endmodule
// wr_stage: takes words from a valid/ready producer and writes them into the FIFO.
// The bug: the write is registered for timing, and the register decides on the full flag of the
// cycle before. A write that makes the FIFO full is followed by one more, which the FIFO drops.
module wr_stage #(parameter bit FIX = 0) (
input logic clk, rst_n, in_valid,
input logic [15:0] in_data,
output logic in_ready,
input logic wfull,
output logic winc,
output logic [15:0] wdata
);
logic v_q;
logic [15:0] d_q;
if (FIX) begin : g_fix // the fix: write in the cycle the flag allows it
assign in_ready = !wfull;
assign winc = in_valid && !wfull;
assign wdata = in_data;
end else begin : g_bad
assign in_ready = !wfull; // bad-example: v_q's write is not counted
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) v_q <= 1'b0;
else begin v_q <= in_valid && !wfull; d_q <= in_data; end
assign winc = v_q;
assign wdata = d_q;
end
endmodule
// flight_rec: a ring buffer that keeps the last DEPTH cycles of a few signals, cheaply, every
// cycle. Nothing is printed until someone calls show(), usually from the check that failed.
module flight_rec #(parameter int DEPTH = 32) (
input logic clk, on,
input logic in_valid, winc, wfull,
input logic [15:0] wdata
);
int unsigned cyc_r [DEPTH];
logic [18:0] sig_r [DEPTH]; // {in_valid, winc, wfull, wdata}
int unsigned wp = 0, cycle = 0;
always @(posedge clk) begin
cycle++;
if (on) begin
cyc_r[wp] = cycle; sig_r[wp] = {in_valid, winc, wfull, wdata};
wp = (wp + 1) % DEPTH;
end
end
function automatic bit holds_drop(); // does the ring still hold a dropped write?
for (int i = 0; i < DEPTH; i++) if (sig_r[i][17] && sig_r[i][16]) return 1;
return 0;
endfunction
task automatic show(input int unsigned last); // print the newest `last` entries, oldest first
for (int k = DEPTH - last; k < DEPTH; k++) begin
int unsigned i;
string note;
i = (wp + k) % DEPTH;
note = (sig_r[i][17] && sig_r[i][16]) ? " <- write dropped" : "";
$display(" cycle %0d: in_valid=%b winc=%b wfull=%b wdata=%0d%s", cyc_r[i], sig_r[i][18],
sig_r[i][17], sig_r[i][16], sig_r[i][15:0], note);
end
endtask
endmodule
// capture_tb: the nightly setting of the previous lesson (rclk 9 ns, 10% single stalls, bursts
// of 1-8), seeds 1 to +SEEDS (200). Two recorders watch the writer, 4 and 32 cycles deep; when
// the consumer finds a number missing, both are frozen and checked, and the first capture is
// printed (+SHOW prints every one). +DUMP=1 dumps everything to full.vcd; +DUMP=2 dumps only
// a 40-cycle window from each precursor (the FIFO going full) to win.vcd. With no plusargs it checks the claims in the lesson.
`timescale 1ns/1ps$ iverilog -g2012 -o c.vvp cap.sv && vvp -n c.vvp
seed 120: word 471 missing, noticed at writer cycle 223389; the 32-cycle recorder holds:
cycle 223376: in_valid=0 winc=1 wfull=0 wdata=465
cycle 223377: in_valid=1 winc=0 wfull=0 wdata=465
cycle 223378: in_valid=1 winc=1 wfull=0 wdata=466
cycle 223379: in_valid=1 winc=1 wfull=0 wdata=467
cycle 223380: in_valid=1 winc=1 wfull=0 wdata=468
cycle 223381: in_valid=1 winc=1 wfull=0 wdata=469
cycle 223382: in_valid=1 winc=1 wfull=0 wdata=470
cycle 223383: in_valid=1 winc=1 wfull=1 wdata=471 <- write dropped
cycle 223384: in_valid=1 winc=0 wfull=0 wdata=472
cycle 223385: in_valid=1 winc=1 wfull=0 wdata=472
cycle 223386: in_valid=0 winc=1 wfull=0 wdata=473
cycle 223387: in_valid=0 winc=0 wfull=0 wdata=473
cycle 223388: in_valid=1 winc=0 wfull=0 wdata=473
cycle 223389: in_valid=0 winc=1 wfull=0 wdata=474
200 seeds: 2 failed; precursor fired 4 times; drop still in the ring: 0 of 2 (4 deep), 2 of 2 (32 deep)
PASS: every failure was captured in the run it happened, with no rerunRead the capture from the bottom up, as you would walk back from a symptom. At 223389 the consumer found 471 missing. Six cycles earlier, at 223383, the writer put 471 on the FIFO while wfull was 1: that write was dropped. The five cycles before it show why: a burst of back-to-back writes, 466 to 470, filled the FIFO, and the stage's registered write could not see the flag in time. The whole story, cause to symptom, is fourteen lines, captured the first time it happened.
The summary line carries the sizing lesson. The detection latency here is six cycles, so the 4-deep ring kept only the last four cycles before the check (223386 to 223389), all after the drop, and held the cause in neither failure. The 32-deep ring held both. The precursor fired four times for two failures: half of its triggers were the FIFO filling with no word lost. Verilator printed the same lines.
$ /usr/bin/time -p vvp -n c.vvp +DUMP=1
200 seeds: 2 failed; precursor fired 4 times; drop still in the ring: 0 of 2 (4 deep), 2 of 2 (32 deep)
real 3.57
$ /usr/bin/time -p vvp -n c.vvp +DUMP=2
200 seeds: 2 failed; precursor fired 4 times; drop still in the ring: 0 of 2 (4 deep), 2 of 2 (32 deep)
real 2.68
$ wc -c full.vcd win.vcd
55149149 full.vcd
27100 win.vcd
55176249 total
$ grep -B1 '^\$dumpon' win.vcd | grep '^#'
#1481425000
#2233825000
#3296415000
#3660565000The full dump of two small modules over 200 runs is 55 MB and costs 37% more run time than no dump (2.61 s). The windowed dump is 27 KB, about 2,000 times smaller, at nearly no extra cost. Its four windows start at 2,233,825 ns and 3,660,565 ns, just before the two failures (seed 120 noticed its gap at 2,233,886 ns), and at two benign full events. The same run, the same failures, the same summary line in all three modes: the capture did not change the run.
One pass of an unchanged regression gave the first wrong cycle of both failures, the data and the flag that explain them, and a small waveform of each, without a rerun. In the lab that is the difference between a fix this week and another week of waiting. You are on the Observe step: the next step, Localise, starts from cycle 223383 instead of from the scoreboard.
The same three structures exist in hardware. Emulators and FPGA logic analysers capture a fixed number of samples around a trigger; chips carry trace buffers that freeze on an error and are read out over the debug port. Their depth and signal count are fixed when the chip is built, which is why the bring-up course and observability by design treat them as design decisions.
It is still useful if each window is small and you can sort them automatically: 4,000 windows of 40 cycles is still far less than a full dump. But it points to a precursor that is too common. Make it more specific, for example "the FIFO goes full while the producer still has a word", or freeze the dump only when the symptom check fires within N cycles of a window, keeping just those windows.
The detection latency is longer than 64 cycles, for example because the wrong word sat in a buffer or memory until it was read (in scoreboards report late, dbglab's unknown word waited 7,434 cycles before the CPU read it); or the recorder is watching the wrong signals, downstream of the cause. Record transactions instead of cycles to reach further back, add a recorder nearer the source, or switch to a precursor trigger.
capture_tb as shown (with afifo and wr_stage from the previous lesson), Icarus Verilog 13.0 (iverilog -g2012 -o c.vvp cap.sv, vvp -n c.vvp) and Verilator 5.048 (verilator --binary --timing --assert -Wno-fatal --top-module capture_tb), 2026-10-18: identical lines; seeds 120 and 197 fail, as in the previous lesson's nightly setting.
/usr/bin/time -p vvp -n c.vvp: 2.61 s with no dump; +DUMP=1 3.57 s and full.vcd 55,149,149 bytes (wc -c); +DUMP=2 2.68 s and win.vcd 27,100 bytes with four windows starting at 1481425000, 2233825000, 3296415000 and 3660565000 ps (grep -B1 '^\$dumpon' win.vcd | grep '^#').
Limits: a small simulated design; the recorder is bench code, not silicon. On-chip trace buffers, emulator triggers and logic analysers apply the same ideas with their own limits on depth and signal count, which this lesson describes but does not run.