Red on Tuesday, green on Wednesday

dbglab's nightly regression runs the random test on fresh seeds every night. About one night in three, one seed goes red with a RAM word that reads xxxxxxxx. The ticket has been reopened four times. (The history is illustrative; every run below is real.) Each time someone reran the seed, saw it fail, then ran the next night's seeds, saw green, and closed it as flaky, cannot reproduce.

Your lead wants the bug fixed or the test removed. Before either, a sharper question: is this test flaky at all, and if it fails one time in N, how do you make it fail every time you ask?

What people usually do, and why it fails

The usual moves are to rerun until green, add an automatic retry, or make the test longer "to stress it". All three treat the failure as noise, and a retry hides it for good. Rerunning a different seed is not rerunning the test: a new seed is new stimulus.

The other common move is to sweep a few dozen seeds, see two failures, and write "fails about 5%". That number is too rough to prove a fix with later.

The method: from flaky to a rate to a repro

1. Check it is flaky. Rerun the same seed, commit and command. If it fails the same way every time, it is not flaky; it is a rare path that some seeds take. That is the usual case, and the good one (determinism and seeds).

2. Name the likely reason for 1 in N. A rare path in the random stimulus; two events that only sometimes land in the same cycle; a clock ratio that only sometimes lines up edges; state that is never initialised and only sometimes read; or the environment (simulator, options, machine).

3. Measure the rate by signature. Sweep a fixed range of seeds, group the FAIL lines by message, and count only your bug's signature. Write the rate with its range.

4. Find the stressor. Ask what every failing seed does that passing seeds do not. Turn the answer into a predicted rate and check it against the measured one.

5. Raise the rate by biasing the stressor. Change the stimulus weights so the mechanism happens more often. Check that the signature did not change.

6. Keep the stressed setting as a named test. It is your repro now, and later your proof.

The picture: one rate, ten samples

Every bar is the same test with the same bug. Seeds 301 to 400 alone would have said 5%; seeds 201 to 300, 16%. Sweep wide before you write a number down.

Do it: measure, predict and raise B4's rate

Is it flaky? Rerun the same seed

Seed 17 fails at cycle 7949 every time, in the same way; the determinism lesson shows the runs. So the test is not flaky. Some seeds take a path that fails, and the question is which path.

Measure by signature, and watch for masking

$ for s in $(seq 1 400); do tb/run.sh random $s | grep FAIL; done > sweep400.txt
$ wc -l < sweep400.txt
     135
$ grep -c xxxxxxxx sweep400.txt
41

135 failures, but only 41 have this bug's signature: an unknown value. The other 94 are a different bug (B3, the beat counter that wraps, from the debug loop), and they mask this one: a seed that hits B3 first stops before it can show B4. To count B4 cleanly, apply B3's known one-line fix in a scratch copy and sweep wider:

$ sed -n 36p rtl/dma_engine.sv                  # scratch copy with B3 fixed
  localparam int CW = 10;           // beat counter width
$ for s in $(seq 1 1000); do tb/run.sh random $s | grep FAIL; done > sweep1000.txt
$ wc -l < sweep1000.txt
     100
$ grep -vc xxxxxxxx sweep1000.txt
0
$ head -3 sweep1000.txt
FAIL test=random seed=16 cycle=686: SCOREBOARD: RAM word 759 (address 00001bdc) read xxxxxxxx, expected 00000000 (last written by DMA run #3)
FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read xxxxxxxx, expected 00000000 (last written by DMA run #4)
FAIL test=random seed=19 cycle=415: SCOREBOARD: RAM word 984 (address 00001f60) read xxxxxxxx, expected 00000000 (last written by DMA run #3)

Read it: 100 failures in 1,000 seeds, every one with an X. The rate is 10.0%, and a 95% interval for 100 in 1,000 is 8.3% to 12.0%. Notice the other thing every line shares: the expected value is always 00000000, and the last writer is always a DMA run. That is the clue to the stressor.

Find the stressor and predict the rate

Expected zero, written by the DMA, means a DMA fill that writes zeros. The random test's DMA chooser shows how often that happens:

1
2
3
4
5
6
7
8
9
10
11
12
13

// snippet: tb/tests.svh, r_start_dma (lines 186-203)
  k = urange(0, 99);
  ...
  else if (k < 25) begin                                             // zero a buffer
    r_fill = 1'b1;
    if (m_fill != 32'h0) csr_wr(CSR_FILL, 32'h0);
    mode = 8'h1 << CTRL_FILL;
  end
  else if (k < 33) begin                                             // pattern fill
    r_fill = 1'b1;
    csr_wr(CSR_FILL, rnd());
    mode = 8'h1 << CTRL_FILL;
  end

A zero fill is 1 DMA run in 100, a pattern fill 8 in 100. A zero fill writes FILL only if the bench's model says FILL is not already zero, so the first zero fill of a run trusts FILL's reset value. A pattern fill writes FILL first. Hypothesis: the failure needs a zero fill before any pattern fill, and FILL does not really reset to zero. That predicts a rate: of the fills in a run, the first is a zero fill with probability 1 in 9, about 11%. The measured 10.0% agrees. (Why FILL is X is the bug itself; finding it is a later part's job.)

The wrong knob: run longer

The intuitive way to stress a random test is to make it longer. The mechanism says that cannot work: only the first fill matters, so more operations add nothing. Check the prediction on seeds 1 to 200:

Four times the operations, three and a half times the wall time, the same rate. A fifth of the operations, the same rate too. The cost moved; the information did not.

The right knob: bias the stressor

In a scratch copy of the bench, widen the zero-fill band from 1 in 100 to 8 in 100, leaving pattern fills at 1 in 100. Prediction: the first fill is a zero fill 8 times in 9, about 89%.

$ sed -n 194p tb/tests.svh                      # scratch copy: zero fills 8 in 100, pattern fills 1 in 100
  else if (k < 32) begin                                             // zero a buffer
$ for s in $(seq 1 200); do tb/run.sh random $s | grep FAIL; done > biased200.txt
$ wc -l < biased200.txt
     177
$ grep -vc xxxxxxxx biased200.txt
0
$ head -2 biased200.txt
FAIL test=random seed=1 cycle=3478: SCOREBOARD: RAM word 169 (address 000012a4) read xxxxxxxx, expected 00000000 (last written by DMA run #17)
FAIL test=random seed=2 cycle=1230: SCOREBOARD: RAM word 817 (address 00001cc4) read xxxxxxxx, expected 00000000 (last written by DMA run #2)

177 of 200, 88.5%, against a prediction of 89%. The signature is unchanged: X, expected zero, written by a DMA fill. Seed 1, which passed before, now fails at cycle 3478. The close match between predicted and measured rates is evidence that you understand the mechanism, before you have opened a waveform.

A smaller case: two events in the same cycle

Many intermittents are collisions: two events that are fine alone and wrong in the same cycle. In this counter of outstanding requests, when issue and retire arrive together the else if order drops the retire. The bench draws the gaps between requests from the seed, so only some seeds collide.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

// outstanding_ctr counts requests issued but not yet retired.
// The bug: when issue and retire land in the same cycle, the if/else order drops the retire.
module outstanding_ctr (
  input  logic       clk, rst_n, issue, retire,
  output logic [3:0] count
);
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n)      count <= '0;
    else if (issue)  count <= count + 4'd1;
    else if (retire) count <= count - 4'd1;    // never reached when issue is also high
endmodule

// The fix, checked on the same stimulus: both events count.
module outstanding_ctr_fixed (
  input  logic       clk, rst_n, issue, retire,
  output logic [3:0] count
);
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) count <= '0;
    else        count <= count + 4'(issue) - 4'(retire);
endmodule

First the view a nightly regression would give you: forty seeds.

$ iverilog -g2012 -o f.vvp flaky_tb.sv
$ vvp -n f.vvp +SWEEP=40
  seed 33 fails: issue and retire together at cycle 45
SWEEP: 1 of 40 seeds fail with gaps of 1 to 40 cycles
$ vvp -n f.vvp +SEED=33
FAIL seed=33 gap=40: count=1 after every request retired (issue and retire together at cycle 45)
$ vvp -n f.vvp
gaps 1-40: 82 of 1000 seeds fail; gaps 1-4: 560 of 1000
PASS: the lost retire is seed-dependent, and short gaps raise its rate

One failure in forty looks like 2.5%; the 1,000-seed rate is 8.2%. Seed 33 fails the same way every time, so again this is a rare path, not a flaky test. The stressor is a collision, so the knob is the spacing of events. The obvious stress, packing requests as close as possible, is not the best one:

With one-cycle gaps all four issues fall within four cycles, so only the retires with the shortest latencies can land on one; collisions get rarer again. The rate peaks when gaps are near the retire latency (2 to 9 cycles). Stress the mechanism, not the volume. Verilator 5.048 prints the same lines, since the bench draws its own numbers.

What the evidence says

Both bugs are deterministic per seed, so neither test is flaky in the strict sense. Each has a measured rate, a named stressor, and a predicted rate that matches. The biased dbglab bench and +GAP=4 are now the repros: they fail in seconds with the same signature, and you will run them again to prove the fix. Reproduce is done; Observe can start from a seed that fails at cycle 1230 instead of one in ten nights.

Traps

Related topics

• Part 1 · Reproduce

• Shrinking a failing test

• The reproduction checklist

• Part 8: intermittent failures

• RTL: debugging with waveforms