"It fails sometimes" is the most expensive sentence in verification. The failure is real until proven otherwise, but nobody can debug it until it is made repeatable. This lesson turns an intermittent failure into a deterministic one, then finds the cause.

Three kinds of "sometimes"

The first question is always: does the same seed on the same build fail every time? If yes, it is not intermittent at all. It is a rare seed, and it can be debugged like any other failure.

The illustrative report

A made-up scenario. The nightly regression of a 4-entry FIFO shows 4 failing seeds out of 40. Yesterday it was 0. The only change was a refactor of the write path. Someone suggests rerunning the failures "to see if they are flaky".

Rerunning a seed-dependent failure gives the same result, so the rerun proves nothing new. The better first step is to count which seeds reach the risky state, and compare that with which seeds fail.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

module fifo4 (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       push,
  input  logic [7:0] wdata,
  input  logic       pop,
  output logic [7:0] rdata,
  output logic       full,
  output logic       empty
);
  logic [7:0] mem [4];
  logic [1:0] wptr, rptr;
  logic [2:0] cnt;
  assign full  = (cnt == 3'd4);
  assign empty = (cnt == 3'd0);
  assign rdata = mem[rptr];
  wire do_push = push && !full;          // one definition, used everywhere
  wire do_pop  = pop  && !empty;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin
      wptr <= '0; rptr <= '0; cnt <= '0;
    end else begin
      if (do_push) begin mem[wptr] <= wdata; wptr <= wptr + 2'd1; end
      if (do_pop)  rptr <= rptr + 2'd1;
      cnt <= cnt + 3'(do_push) - 3'(do_pop);
    end
endmodule

The bench runs 40 seeds in one simulation. Each seed is set with srandom(seed) on the running thread's std::process handle, so the stream is the same every run on the same build (see Random Stability and Seeding).

With the correct FIFO it passes, and reports that 9 of the 40 seeds reach the state "full, with push and pop in the same cycle". Now apply the refactor that caused the report: the write path uses its own copy of the push condition.

// snippet: the refactor, write path only
if (push && (!full || pop)) begin mem[wptr] <= wdata; wptr <= wptr + 2'd1; end
// the count still uses do_push = push && !full

With this change the bench prints FAIL: 4 of 40 seeds failed: 1 16 21 22 (Verilator 5.048). The pattern is the clue:

• Every failing seed is one of the 9 that reach the corner. No seed fails without it.

• Not every corner seed fails. The pointer and the count disagree after the corner, but the data goes wrong only if another push lands before the end of the test.

• So the bug is reachable on about one seed in four, and visible on about one in ten. That is what "intermittent" usually means: rare, not random.

Root cause: two places compute "is this push accepted". At full with a pop, the write path accepts the push and moves the pointer, but the count does not. The next push overwrites the oldest entry. The fix is the one in the listing: define do_push once and use it everywhere.

Make it deterministic, then shrink it

1. Pin seed, build, tool version and plusargs. Rerun once. If it passes now, you have a stability or race problem, not a rare seed.

2. Find the first wrong event, not the end-of-test message. Here: the first read that returns the wrong data.

3. Shrink: cut the test length to just past the first wrong event, and turn off agents that do not matter.

4. Correlate: which seeds reach the suspected state? If every failing seed reaches it and no passing seed shows the symptom, you have strong evidence.

5. Add the state as a coverage point, so the regression reports how often the corner is reached.

When the same seed does not reproduce

If a failure disappears when a $display is added, or appears only on one simulator, suspect a race. Two processes woken by the same event run in an order the LRM leaves undefined (IEEE 1800-2017, 4.7). The classic case is a testbench driving with a blocking assignment at the clock edge:

1
2
3
4
5

// snippet: a testbench race and its fix
always @(posedge clk) valid = next_valid;            // races with the DUT's always_ff reading valid
always @(posedge clk) valid <= next_valid;           // fix 1: nonblocking, updated after all reads
initial forever @(negedge clk) valid = next_valid;   // fix 2: drive half a cycle away from the edge
// fix 3: drive through a clocking block with an output skew

Whether the DUT sees the old or the new valid depends on which always runs first. Many simulators keep a stable order for one build, so the failure looks repeatable until an unrelated edit changes the order. See Anatomy of a TB/RTL Race for the scheduler detail.

Random stability causes a third kind. Each thread and object has its own random generator, seeded from its parent when it is created (IEEE 1800-2017, 18.14). Adding one randomize() call or one new object before others can shift every stream after it. The same seed then takes a different path. The fix is not to rerun the old seed, but to accept that the old seed is gone and keep the regression wide enough to find the corner again.

Related topics

• Previous: Escaped Bug Analysis

• Next: Metrics That Lie

• Seeds & Reproducibility

• Hunting X's and Races