A stream block passes words from a producer clock to a consumer clock through an 8-word async FIFO. Once in a while the consumer finds a sequence number missing. The nightly setting fails about one seed in a hundred: two in the last two hundred, seeds 120 and 197, both reproducible.
You have a repro, but each experiment costs a regression. Before you start guessing, the useful question is: what change to the stimulus makes this bug fail most often per hour of simulation?
The usual answers are "run more seeds" and "run longer". Both buy failures at the same price per failure, so the next experiment is no faster. The other usual answer is to turn every knob to its maximum at once. That may raise the rate, but you learn nothing about the mechanism, and you may create a different failure, or a hang, that you then mistake for the first.
Making a flaky test fail showed one case where running longer added nothing, because only the first fill of a run mattered. This lesson measures the other knobs on a bug where time does matter, and puts a number on each.
Name the state. From the mechanism, or from what the failing seeds share, write down the state the design must reach. Here: the FIFO goes full while the producer is still sending.
Score stresses in failures per unit of cost. Count failures per simulated millisecond, or per CPU hour. Per-run rates reward longer runs that cost more.
Change one knob at a time (one change at a time), over the same seeds, and keep the table.
Try the four usual knobs: run length; the amount and the shape of back-pressure; clock ratio and phase; and a targeted bias of the stimulus toward the state.
Check the signature of every failure in a stressed run. A new message means a new bug, not a higher rate.
Keep the best setting as a named test. It is the repro now and the proof of the fix later.
Knob | What it buys | When it pays |
|---|---|---|
Longer runs, more seeds | More of the same chances, at the same price | Never per unit of cost; only when the state takes long to build up |
More back-pressure | More cycles where data waits | When the state needs occupancy |
Back-pressure in bursts | Long waits, which single stalls rarely make | When the state needs a buffer to fill or drain completely |
Clock ratio and phase | Different alignments of two clocks | When the state needs one side faster, or a particular edge alignment |
Targeted bias | More of the stimulus that builds the state | When you know which operation builds it |
The design: a writer stage registers its FIFO write for timing, deciding on the full flag of the cycle before. A write that makes the FIFO full is followed by one more, which the FIFO drops. The fix writes in the cycle the flag allows it. The bench sends numbered words in bursts from a 10 ns clock and a consumer pops them on its own clock with stalls; every knob is a plusarg, and with none it runs the lesson's table and checks it.
// afifo: a compact version of dbglab's async FIFO (two-flop Gray pointer synchronisers, inlined), here 8 words deep
module afifo #(parameter int DW = 16, parameter int AW = 3) (
input logic wclk, wrst_n, winc,
input logic [DW-1:0] wdata,
output logic wfull,
input logic rclk, rrst_n, rinc,
output logic [DW-1:0] rdata,
output logic rempty
);
localparam logic [AW:0] TOP2 = 3 << (AW - 1);
logic [DW-1:0] mem [1 << AW];
logic [AW:0] wbin, wbin_n, wgray, wgray_n, rq1, rptr_w;
logic [AW:0] rbin, rbin_n, rgray, rgray_n, wq1, wptr_r;
assign wbin_n = wbin + (winc & ~wfull);
assign wgray_n = (wbin_n >> 1) ^ wbin_n;
always_ff @(posedge wclk or negedge wrst_n)
if (!wrst_n) begin wbin <= '0; wgray <= '0; wfull <= 1'b0; rq1 <= '0; rptr_w <= '0; end
else begin
wbin <= wbin_n; wgray <= wgray_n; wfull <= (wgray_n == (rptr_w ^ TOP2));
rq1 <= rgray; rptr_w <= rq1;
end
always_ff @(posedge wclk) if (winc && !wfull) mem[wbin[AW-1:0]] <= wdata;
assign rbin_n = rbin + (rinc & ~rempty);
assign rgray_n = (rbin_n >> 1) ^ rbin_n;
always_ff @(posedge rclk or negedge rrst_n)
if (!rrst_n) begin rbin <= '0; rgray <= '0; rempty <= 1'b1; wq1 <= '0; wptr_r <= '0; end
else begin
rbin <= rbin_n; rgray <= rgray_n; rempty <= (rgray_n == wptr_r);
wq1 <= wgray; wptr_r <= wq1;
end
assign rdata = mem[rbin[AW-1:0]];
endmodule
// wr_stage: takes words from a valid/ready producer and writes them into the FIFO.
// The bug: the write is registered for timing, and the register decides on the full flag of the
// cycle before. A write that makes the FIFO full is followed by one more, which the FIFO drops.
module wr_stage #(parameter bit FIX = 0) (
input logic clk, rst_n, in_valid,
input logic [15:0] in_data,
output logic in_ready,
input logic wfull,
output logic winc,
output logic [15:0] wdata
);
logic v_q;
logic [15:0] d_q;
if (FIX) begin : g_fix // the fix: write in the cycle the flag allows it
assign in_ready = !wfull;
assign winc = in_valid && !wfull;
assign wdata = in_data;
end else begin : g_bad
assign in_ready = !wfull; // bad-example: v_q's write is not counted
always_ff @(posedge clk or negedge rst_n)
if (!rst_n) v_q <= 1'b0;
else begin v_q <= in_valid && !wfull; d_q <= in_data; end
assign winc = v_q;
assign wdata = d_q;
end
endmodule
// stress_tb: a producer on wclk (10 ns) sends numbered words in bursts; a consumer on rclk pops
// them with stalls and checks the numbers. A run (one seed) fails if a number goes missing.
// Knobs, as plusargs: RCLK=<ps> consumer clock period (default 7000); STALL=<pct> share of
// consumer cycles stalled (10); SRUN=<n> longest stall, 1 = single-cycle stalls (1);
// BURST=<n> longest producer burst (8); BURSTS=<n> bursts per run (200); SEEDS=<n> (200);
// RPHASE=<ps> delays rclk's first edge; FIX=1 uses the fixed wr_stage. With no plusargs it runs the lesson's table and checks it.
`timescale 1ns/1ps$ iverilog -g2012 -o s.vvp stress_tb.sv && vvp -n s.vvp
nightly: rclk 9.0 ns, 10% single stalls, bursts of 1-8: 2 of 200 runs fail (1.0%), 3734 us simulated, 0.54 failures per simulated ms
longer: 800 bursts per run: 6 of 200 runs fail (3.0%), 14537 us simulated, 0.41 failures per simulated ms
more stalls: 20% single stalls: 38 of 200 runs fail (19.0%), 3734 us simulated, 10.18 failures per simulated ms
bursty stalls: 10% in runs of 1-8: 196 of 200 runs fail (98.0%), 3750 us simulated, 52.27 failures per simulated ms
clock ratio: rclk 11.0 ns: 123 of 200 runs fail (61.5%), 3736 us simulated, 32.92 failures per simulated ms
targeted: bursts of 1-16: 194 of 200 runs fail (97.0%), 5339 us simulated, 36.34 failures per simulated ms
fixed wr_stage, bursty stalls and bursts of 1-16: 0 of 200 runs fail (0.0%), 5460 us simulated, 0.00 failures per simulated ms
PASS: shaped stress raises the failures per simulated ms; the fix survives itRead it row by row against the state "FIFO full while the producer sends".
Longer runs quadrupled the cost and tripled the per-run rate (1.0% to 3.0%), so the yield per millisecond did not rise (0.54 to 0.41). The same holds further out: +BURSTS=3200 failed 26 of 200 runs at 0.45 per ms, and 1,000 nightly seeds gave 13 failures at 0.70 per ms. This bug has a steady chance per cycle; time buys it at a fixed price.
Twice the stalls gave 19 times the yield. Stalls let words pile up, which is the state.
The same 10% of stalls in bursts gave 97 times the yield and failed 196 runs in 200. An 8-word FIFO fills when the consumer stops for several cycles in a row; single-cycle stalls, however many, rarely do that.
A slower consumer clock, or longer producer bursts from the other side, fill the FIFO on every long burst: only bursts longer than the FIFO's slack can fill it.
The fix survived the hardest combined stress, 200 runs, which is a far stronger statement than 200 nightly runs (the statistics lesson puts a number on it).
Verilator 5.048 printed the same seven lines in 2.6 s. Every failure in every row reports the same signature, word N missing, and the fixed design never fails, so the stresses raised the rate of this bug rather than adding another.
The clock-ratio row used one ratio. A sweep shows more, and holding the ratio while moving only the consumer clock's first edge (+RPHASE) shows a trap:
$ for r in 8000 9000 9500 10000 10500 11000 12000; do for p in 0 2500; do
> vs/Vstress_tb +RCLK=$r +RPHASE=$p | head -1; done; done
rclk=8000 stall=10 srun=1 burst=8 bursts=200 fix=0: 0 of 200 runs fail (0.0%), 3734 us simulated, 0.00 failures per simulated ms
rclk=8000 stall=10 srun=1 burst=8 bursts=200 fix=0: 0 of 200 runs fail (0.0%), 3734 us simulated, 0.00 failures per simulated ms
rclk=9000 stall=10 srun=1 burst=8 bursts=200 fix=0: 2 of 200 runs fail (1.0%), 3734 us simulated, 0.54 failures per simulated ms
rclk=9000 stall=10 srun=1 burst=8 bursts=200 fix=0: 3 of 200 runs fail (1.5%), 3734 us simulated, 0.80 failures per simulated ms
rclk=9500 stall=10 srun=1 burst=8 bursts=200 fix=0: 13 of 200 runs fail (6.5%), 3734 us simulated, 3.48 failures per simulated ms
rclk=9500 stall=10 srun=1 burst=8 bursts=200 fix=0: 10 of 200 runs fail (5.0%), 3734 us simulated, 2.68 failures per simulated ms
rclk=10000 stall=10 srun=1 burst=8 bursts=200 fix=0: 114 of 200 runs fail (57.0%), 3736 us simulated, 30.51 failures per simulated ms
rclk=10000 stall=10 srun=1 burst=8 bursts=200 fix=0: 20 of 200 runs fail (10.0%), 3734 us simulated, 5.36 failures per simulated ms
rclk=10500 stall=10 srun=1 burst=8 bursts=200 fix=0: 77 of 200 runs fail (38.5%), 3735 us simulated, 20.62 failures per simulated ms
rclk=10500 stall=10 srun=1 burst=8 bursts=200 fix=0: 82 of 200 runs fail (41.0%), 3735 us simulated, 21.95 failures per simulated ms
rclk=11000 stall=10 srun=1 burst=8 bursts=200 fix=0: 123 of 200 runs fail (61.5%), 3736 us simulated, 32.92 failures per simulated ms
rclk=11000 stall=10 srun=1 burst=8 bursts=200 fix=0: 141 of 200 runs fail (70.5%), 3736 us simulated, 37.74 failures per simulated ms
rclk=12000 stall=10 srun=1 burst=8 bursts=200 fix=0: 196 of 200 runs fail (98.0%), 3741 us simulated, 52.39 failures per simulated ms
rclk=12000 stall=10 srun=1 burst=8 bursts=200 fix=0: 196 of 200 runs fail (98.0%), 3741 us simulated, 52.39 failures per simulated msEach pair is one ratio at phase 0 and at 2.5 ns. Away from 1:1 the phase moves the rate by a few points (61.5% against 70.5% at 11 ns is at the edge of sampling noise). At exactly 10 ns against 10 ns it moves it from 57% to 10%. At phase 0 every consumer edge lands at the same instant as a producer edge, one fixed alignment held for the whole run, which two real oscillators never do. Treat a result at exactly 1:1 and phase 0 as a property of the bench setup, not of the design. Icarus gives the same two numbers. Away from that point the rate climbs steadily as the consumer slows: a ratio sweep turns "sometimes" into a curve.
The best single knob, stalls in bursts, turned a one-in-a-hundred failure into 196 in 200 at the same cost per run, and named the mechanism on the way: the bug needs a full FIFO, and a full FIFO needs the consumer to stop for several cycles. The stressed setting is now the repro: with +SRUN=8 +SHOW, seed 1 reports word 178 missing 3,587 ns into the simulation, instead of waiting for seed 120 of the nightly setting. The fix has already passed it 200 times.
This is still the Reproduce step. Next, the rare event has to be captured when it does happen, which matters most where you cannot rerun at all.
How much longer the runs took. In the table above, four times the bursts cost four times the simulated time and gave 3.0% of runs, which is 0.41 failures per simulated millisecond against 0.54 for the default: no better at all. Rates must be compared per unit of cost, or every experiment that just runs longer looks like progress.
stress_tb as shown, Icarus Verilog 13.0 (iverilog -g2012 -o s.vvp stress_tb.sv, vvp -n s.vvp, 22 s) and Verilator 5.048 (verilator --binary --timing --assert -Wno-fatal --top-module stress_tb, 2.6 s), 2026-10-18: the seven-row table, identical in both. +SRUN=8 +SEEDS=3 +SHOW: seed 1 misses word 178 at 3587.0 ns. Also +RCLK=9000 +SEEDS=1000 (13 of 1000, 18636 us), +RCLK=9000 +BURSTS=3200 (26 of 200, 57696 us), +RCLK=<ps> +RPHASE=<ps> for 8000 to 12000 and phases 0 and 2500, and +RCLK=9000 +SHOW (seeds 120 and 197 in both simulators).
Each row runs seeds 1 to 200 one after another in one simulation, so a row's rclk phase is wherever the previous row left it; standalone runs start rclk at the plusarg phase.
Limits: zero-delay simulation of a small design. The numbers show how much each kind of stress matters for this mechanism; another bug needs another state, and the ranking can change. Coincident clock edges at phase 0 are a simulation artefact.