It failed last night, and it passes this morning

The nightly regression reports that dbglab's random test failed. You open a terminal and run the test:

$ tb/run.sh random
dbglab: test=random seed=1
PASS test=random seed=1 cycles=19972 time=199716 ns

It passes. Is the bug gone, was it a fluke, or did you just run a different test?

What people usually do, and why it fails

The common reaction is to rerun a few times, see green, and mark the failure as flaky. But that rerun was not the failing run. The nightly used seed 10; the command above used the default, seed 1. A different seed is different stimulus: a different test that happens to share a name.

The same mistake comes in other forms. Rerunning on today's commit instead of last night's. Rerunning in the other simulator. Rerunning with a debug switch that changes the length of the test. Each one changes an input of the run, and each one can turn a real failure into a pass that means nothing.

The method: name every input, then fix it

A simulation is a program. Given the same inputs, a well-built bench produces the same run, cycle for cycle. Your job at the Reproduce step is to know what those inputs are:

1. The seed. It decides every random choice. Always pass it on the command line and print it in the first log line, as dbglab does (dbglab: test=random seed=10).

2. The test and its plusargs. +TEST, and anything else read with $value$plusargs, such as dbglab's +OPS (the length of the random test).

3. The commit, plus any uncommitted edits. Different RTL or bench code is a different program.

4. The simulator and its version. Simulators differ in scheduling, in X handling and in their built-in random generators.

5. Options and environment. Defines, compile switches, and environment variables your scripts read, such as dbglab's XRAND.

There is a sixth input you do not want: the order in which the simulator runs processes that wake at the same time. The language leaves that order open. A good bench is written so the order cannot change the result. Then check the claim: run the same command twice and compare the logs.

The picture

Do it: dbglab repeats itself

Same inputs, same run

$ tb/run.sh random 10 > /dev/null; cp build/icarus-random-10.log run1.log
$ tb/run.sh random 10 > /dev/null; cp build/icarus-random-10.log run2.log
$ cmp run1.log run2.log && echo identical
identical
$ cat run1.log
dbglab: test=random seed=10
FAIL test=random seed=10 cycle=4895: SCOREBOARD: RAM word 808 (address 00001ca0) read d5202ff9, expected b6c3311c (last written by DMA run #29)
FATAL: tb/dbglab_tb.sv:57: test failed
       Time: 48946000  Scope: dbglab_tb.fail
$ tb/run.sh random 10 verilator > /dev/null
$ diff build/icarus-random-10.log build/verilator-random-10.log
3,4c3,5
< FATAL: tb/dbglab_tb.sv:57: test failed
<        Time: 48946000  Scope: dbglab_tb.fail
---
> [48946000] %Fatal: dbglab_tb.sv:57: Assertion failed in dbglab_tb.fail: test failed
> %Error: tb/dbglab_tb.sv:57: Verilog $stop
> Aborting...

Read it from the top. Two Icarus runs give byte-identical logs. Verilator gives the same FAIL line, and the only lines that differ are each simulator's own way of reporting $fatal; both stop at the same simulation time, 48946000. Comparing whole logs, not just the verdict, is what makes this a proof.

Change one input, get another run

$ tb/run.sh random 11
dbglab: test=random seed=11
PASS test=random seed=11 cycles=18509 time=185086 ns
$ tb/run.sh random 13
dbglab: test=random seed=13
FAIL test=random seed=13 cycle=3701: SCOREBOARD: RAM word 250 (address 000013e8) read f42b063a, expected c6278de6 (last written by DMA run #22)
$ tb/run.sh random 10 icarus +OPS=200
dbglab: test=random seed=10
PASS test=random seed=10 cycles=2528 time=25276 ns
$ git checkout -q bc2bb34 && tb/run.sh random 10      # commit 12, the day before the bug
dbglab: test=random seed=10
PASS test=random seed=10

• Seed 11 draws different stimulus and never hits the bug. Seed 13 hits it at a different cycle, on a different word.

• `+OPS=200` keeps the seed but stops after 200 operations, before the failing DMA run.

• Commit 12 is the same seed and command on older code. It passes because the bug arrives in commit 13. (The PASS line has no cycle count yet; that arrives in commit 16.)

The simulator is an input too. dbglab's seed 17 fails on an unknown value:

$ tb/run.sh random 17
dbglab: test=random seed=17
FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read xxxxxxxx, expected 00000000 (last written by DMA run #4)
$ tb/run.sh random 17 verilator
dbglab: test=random seed=17
PASS test=random seed=17 cycles=18442 time=184416 ns
$ XRAND=1 tb/run.sh random 17 verilator
dbglab: test=random seed=17
FAIL test=random seed=17 cycle=7949: SCOREBOARD: RAM word 816 (address 00001cc0) read 5d88e2dd, expected 00000000 (last written by DMA run #4)

Icarus has an X state; Verilator, by default, starts every flip-flop at 0, which here happens to be the right value. With XRAND=1 dbglab's script asks Verilator for random initial values, seeded by the same +SEED, and the failure comes back. Same seed, same commit, three results: the simulator and its options belong in every reproduction. (Why the value is unknown is a story for Part 5.)

Where determinism comes from in dbglab's bench

dbglab does not use the simulator's random functions. It has its own generator, a xorshift, and only the main test process calls it:

1
2
3
4
5
6
7
8
9
10

// snippet: tb/dbglab_tb.sv, the bench's random generator and how the seed starts it
  function automatic int unsigned rnd();
    rng = rng ^ (rng << 13);
    rng = rng ^ (rng >> 17);
    rng = rng ^ (rng << 5);
    return rng;
  endfunction
...
    rng = seed * 32'h9E37_79B9 + 32'h7F4A_7C16;   // spread small seeds apart
    if (rng == 0) rng = 1;                           // xorshift must not start at 0

Three shifts and three XORs are plain integer arithmetic, so every simulator computes the same numbers. Compare the built-in $urandom, called with no seed:

1
2
3
4

// Three calls to $urandom with no seed given: what does each simulator return?
module urandom_demo;
  initial $display("urandom: %08h %08h %08h", $urandom, $urandom, $urandom);
endmodule

$ iverilog -g2012 -o u.vvp urandom_demo.sv && vvp -n u.vvp
urandom: 92153524 40895e81 0484d609
$ verilator --binary -Wno-fatal --top-module urandom_demo -Mdir vl urandom_demo.sv > /dev/null && vl/Vurandom_demo
urandom: fa43d709 c004b34e 719ca9ad
...
$ vl/Vurandom_demo +verilator+seed+7
urandom: 9666691b a2c1a89c c57479fe
...

Each simulator is repeatable on its own, but the numbers differ between them, and each has its own way to set the seed. That is fine for a bench that will only ever run on one tool. dbglab's own generator is why seed 10 fails at cycle 4895 in both.

Random stability: a new draw can move other numbers

Next hazard: you add one random call to one part of the bench. Does every other part still get the same numbers? SystemVerilog intends each process to have its own generator ("random stability"), but simulators differ:

1
2
3
4
5
6
7
8
9
10
11
12
13

// Random stability: does one extra draw in process A change what process B draws?
module stability #(parameter bit EXTRA = 1'b0);
  int unsigned a1, b1, b2;
  initial begin                                  // process A
    a1 = $urandom;
    if (EXTRA) void'($urandom);                  // a new draw, added later in A only
  end
  initial begin                                  // process B
    #1 b1 = $urandom;
    b2 = $urandom;
    $display("EXTRA=%0d  B draws: %08h %08h", EXTRA, b1, b2);
  end
endmodule

$ iverilog -g2012 -Pstability.EXTRA=0 -o s0.vvp stability.sv && vvp -n s0.vvp
EXTRA=0  B draws: 40895e81 0484d609
$ iverilog -g2012 -Pstability.EXTRA=1 -o s1.vvp stability.sv && vvp -n s1.vvp
EXTRA=1  B draws: 0484d609 31f05663
$ verilator --binary -Wno-fatal -GEXTRA=0 --top-module stability -Mdir vl0 stability.sv > /dev/null && vl0/Vstability
EXTRA=0  B draws: fa43d709 c004b34e
...
$ verilator --binary -Wno-fatal -GEXTRA=1 --top-module stability -Mdir vl1 stability.sv > /dev/null && vl1/Vstability
EXTRA=1  B draws: fa43d709 c004b34e
...

In Icarus all processes share one stream, so A's extra draw shifts B's numbers by one. Verilator keeps B's numbers unchanged. On a simulator like Icarus, adding a debug $urandom anywhere can change the stimulus everywhere, and the failure you were chasing can vanish the moment you instrument it.

Process order: two processes, one generator

The last hazard is the one in the figure. dbglab's generator is safe because only one process calls it. Here two processes call the same generator on the same clock edge:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

// bad-example: two processes share one generator, so who gets which number depends on scheduling
module shared_rng_tb;
  logic clk = 1'b0;
  int unsigned rng = 32'h1234_5678;           // ONE generator state for the whole bench
  function automatic int unsigned rnd();      // xorshift32, as in dbglab's bench
    rng = rng ^ (rng << 13);
    rng = rng ^ (rng >> 17);
    rng = rng ^ (rng << 5);
    return rng;
  endfunction
  int unsigned len_a, len_b;
  always @(posedge clk) len_a = 1 + rnd() % 64;   // stimulus for port A
  always @(posedge clk) len_b = 1 + rnd() % 64;   // stimulus for port B
  initial
    repeat (3) begin
      #5 clk = 1'b1;
      #1 $display("port A len=%0d  port B len=%0d", len_a, len_b);
      #4 clk = 1'b0;
    end
endmodule

Run it in both simulators, then swap the two always lines (shared_ba.sv) and run again:

The same six numbers come out every time, but which port gets which depends on the order the simulator runs the two processes. Icarus follows the source order here; Verilator's order differs, and not even the same way on every edge. A harmless-looking edit, such as moving a block or adding a file, would change the stimulus of both ports. The fix is to give each stream its own generator:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

// Each process owns its generator, seeded from the test seed and its own name.
// The numbers a process draws no longer depend on which process the simulator runs first.
module own_rng_tb;
  logic clk = 1'b0;
  int unsigned seed = 10;
  function automatic int unsigned xs(input int unsigned s);   // one xorshift32 step
    s = s ^ (s << 13);
    s = s ^ (s >> 17);
    s = s ^ (s << 5);
    return s;
  endfunction
  int unsigned rng_a, rng_b, len_a, len_b;
  initial begin
    rng_a = seed * 32'h9E37_79B9 + 32'h0000_00A1;             // stream for port A
    rng_b = seed * 32'h9E37_79B9 + 32'h0000_00B2;             // stream for port B
  end
  always @(posedge clk) begin rng_a = xs(rng_a); len_a = 1 + rng_a % 64; end
  always @(posedge clk) begin rng_b = xs(rng_b); len_b = 1 + rng_b % 64; end

  // Check: recompute both streams on their own, in a fixed order, and compare every cycle.
  int unsigned ref_a, ref_b;
  initial begin
    ref_a = seed * 32'h9E37_79B9 + 32'h0000_00A1;
    ref_b = seed * 32'h9E37_79B9 + 32'h0000_00B2;
    repeat (3) begin
      #5 clk = 1'b1;
      #1 ref_a = xs(ref_a); ref_b = xs(ref_b);
      $display("port A len=%0d  port B len=%0d", len_a, len_b);
      if (len_a != 1 + ref_a % 64 || len_b != 1 + ref_b % 64) $fatal(1, "FAIL: stream depends on process order");
      #4 clk = 1'b0;
    end
    $display("PASS: each port's lengths come from its own stream");
    $finish;
  end
endmodule

$ iverilog -g2012 -o o.vvp own_rng_tb.sv && vvp -n o.vvp
port A len=52  port B len=8
port A len=10  port B len=60
port A len=39  port B len=26
PASS: each port's lengths come from its own stream
...

With the two always lines swapped, and in Verilator in both orders, the output is the same three lines and PASS.

What the evidence says

dbglab's seed 10 is a real reproduction: the same command gives the same log in two simulators, and changing the seed, a plusarg, the commit or the simulator options changes the result in a way you can explain. So the morning's PASS proved nothing; the right command is tb/run.sh random 10 at last night's commit, and it fails. The small benches show the three ways a bench can lose this: the simulator's own generator, a draw added in one place shifting numbers elsewhere, and a generator shared between processes.

Traps

Related topics

• Part 1 · Reproduce

• Next: recording everything to replay a failure

• The debug loop

• SystemVerilog: seeds and reproducibility

• SystemVerilog: randomisation debug

• RTL: the blocking vs non-blocking race