The nightly report says dma_rand_test failed with a scoreboard mismatch. You rerun the same test on your desk and it passes. You rerun it ten more times; it passes ten more times. A colleague says "must be flaky" and adds it to the rerun-on-fail list. Two weeks later the same mismatch shows up in a customer's silicon. The failure was never flaky: your rerun simply was not the same run. What exactly makes a run repeatable?

The idea: a random test is a recipe, and the seed is only one ingredient

A constrained-random test does not choose its stimulus; the simulator's random number generator (RNG) does, starting from one integer, the seed. The same seed gives the same sequence of random choices, so the same lengths, keys and data words. Change the seed and every choice changes. That is what makes random testing powerful (each seed is a new test for free) and what makes a failure hard to bring back: if you do not know the seed, you cannot get the run back.

Think of a baking recipe that says "some flour, some sugar". To bake the same cake twice you need the exact amounts (the seed), but also the same recipe (the test source), the same oven (the build and the simulator version) and the same oven settings (the plusargs). Change any of them and you bake a different cake, even with the same amounts. A replay bundle is that full list, recorded for every run:

The picture: replay first, then decide what kind of failure it is

Build it: seeds on the course bench

The seed switch

Every simulator has its own seed switch. Questa takes -sv_seed <n>, Xcelium takes -svseed <n>, VCS takes +ntb_random_seed=<n>, and each also has a "pick one for me" mode that prints the value it chose. The course's Verilator build reads a runtime plusarg, +verilator+seed+<n>, so one compiled binary runs any seed. The per-tool details and the launcher policy are in Seed Control and Regression Replay.

Here is dma_rand_test (three random jobs, LEN 1 to 16) run from one build with no seed, with seed 1 twice, and with seed 2:

$ ./Vtop +UVM_TESTNAME=dma_rand_test
UVM_INFO dma_tb.sv(731) @ 0: uvm_test_top [JOB] key=90fdd8fd len=10 irq_en=1
UVM_INFO dma_tb.sv(731) @ 645000: uvm_test_top [JOB] key=0e564b57 len=1 irq_en=1
UVM_INFO dma_tb.sv(731) @ 885000: uvm_test_top [JOB] key=5a1793f4 len=3 irq_en=1
UVM_INFO dma_tb.sv(411) @ 1305000: uvm_test_top.env.sb [SB] 14 words matched, 0 mismatched

$ ./Vtop +UVM_TESTNAME=dma_rand_test +verilator+seed+1      (run twice: identical)
UVM_INFO dma_tb.sv(731) @ 0: uvm_test_top [JOB] key=8f8a862e len=3 irq_en=1
UVM_INFO dma_tb.sv(731) @ 475000: uvm_test_top [JOB] key=de0dd8aa len=15 irq_en=1
UVM_INFO dma_tb.sv(731) @ 1305000: uvm_test_top [JOB] key=23c8fd76 len=7 irq_en=1
UVM_INFO dma_tb.sv(411) @ 1915000: uvm_test_top.env.sb [SB] 25 words matched, 0 mismatched

$ ./Vtop +UVM_TESTNAME=dma_rand_test +verilator+seed+2
UVM_INFO dma_tb.sv(731) @ 0: uvm_test_top [JOB] key=01bf9c54 len=11 irq_en=1
UVM_INFO dma_tb.sv(731) @ 625000: uvm_test_top [JOB] key=4f43338b len=13 irq_en=1
UVM_INFO dma_tb.sv(731) @ 1365000: uvm_test_top [JOB] key=44ccdb75 len=16 irq_en=1
UVM_INFO dma_tb.sv(411) @ 2445000: uvm_test_top.env.sb [SB] 40 words matched, 0 mismatched

• No seed is still a seed. The run without the plusarg is the bench's familiar 14-word run; on this build it repeats every time. Other tools pick a fixed default too, but the value differs between tools and sometimes between versions, so "no seed" is never a replay record.

• One seed is a whole test. Seed 2 moved 40 words and hit LEN=16, a value the default seed never reaches. That is why a nightly runs many seeds of the same test.

• Same seed, same run. Seed 1 run twice gives identical lines, down to the timestamps.

Replay a failure, and verify the fix on the same stimulus

The mismatch build of the bench (+define+DMA_BUG_WORD6, which corrupts the sixth word of a job) failed dma_rand_test on seed 16. Replaying the bundle gives the same first error at the same time, and running the same seed on the fixed build gives the same jobs with no mismatches. Because the bug lives in the DUT, not in the testbench, it does not change the testbench's random draws: the same seed is the same stimulus on both builds, which is what makes it a fair fix check.

$ ./Vtop_bug +UVM_TESTNAME=dma_rand_test +verilator+seed+16
UVM_INFO dma_tb.sv(731) @ 0: uvm_test_top [JOB] key=3c92310b len=7 irq_en=1
UVM_ERROR dma_tb.sv(396) @ 345000: uvm_test_top.env.sb [SB_MISMATCH] expected 4735a210, got b8ca5def
UVM_INFO dma_tb.sv(731) @ 445000: uvm_test_top [JOB] key=71d48f2b len=2 irq_en=1
UVM_INFO dma_tb.sv(731) @ 715000: uvm_test_top [JOB] key=c6550b9f len=10 irq_en=1
UVM_ERROR dma_tb.sv(396) @ 1295000: uvm_test_top.env.sb [SB_MISMATCH] expected e76189d2, got 189e762d
UVM_INFO dma_tb.sv(411) @ 1665000: uvm_test_top.env.sb [SB] 17 words matched, 2 mismatched

$ ./Vtop_fixed +UVM_TESTNAME=dma_rand_test +verilator+seed+16
UVM_INFO dma_tb.sv(731) @ 0: uvm_test_top [JOB] key=3c92310b len=7 irq_en=1
UVM_INFO dma_tb.sv(731) @ 445000: uvm_test_top [JOB] key=71d48f2b len=2 irq_en=1
UVM_INFO dma_tb.sv(731) @ 715000: uvm_test_top [JOB] key=c6550b9f len=10 irq_en=1
UVM_INFO dma_tb.sv(411) @ 1665000: uvm_test_top.env.sb [SB] 19 words matched, 0 mismatched

A testbench change is different. Adding a randomized field, a new sequence or a randomize() call can shift which random numbers later objects receive, so after a bench edit an old seed may no longer reach the failing case. That is why the bundle records the build, and why a fix check reruns the failing seed before anything else in the bench changes.

Random stability: what the standard promises, and what to verify

IEEE 1800 (section 18.14) gives every thread and every object its own RNG, seeded from its parent when it is created, so that adding a random call in one place does not disturb the numbers drawn elsewhere. This random stability is what lets you add a test without changing every other test's stimulus. Tools implement it to different depths, so test it on yours. The model below draws in two threads; a scratch rebuild with -DEXTRA adds one $urandom in thread a before its object is created.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

module top;
  class job;
    rand bit [7:0] len;
  endclass
  initial begin : a
    job j;
`ifdef EXTRA
    int unused = $urandom;      // one extra draw in thread a, before new()
`endif
    j = new;
    void'(j.randomize());
    $display("a: len=%0d", j.len);
  end
  initial begin : b
    job j = new;
    #1;
    void'(j.randomize());
    $display("b: len=%0d", j.len);
  end
endmodule

scratch runs, Verilator 5.048:
  plain build:   a: len=58   b: len=43
  -DEXTRA:       a: len=58   b: len=43

Thread b is unchanged, as the standard requires. By the standard, thread a's object should change, because it is seeded from a thread RNG that has moved on; here it did not, so this Verilator build seeds objects differently from the LRM's description. Neither result is a problem for replay (the same build always repeats), but it shows why "my edit should not change other tests' stimulus" is a claim to check on your tool, not to assume.

Seed policy: fixed where you gate, fresh where you explore

• Smoke and merge-gate runs use fixed seeds. A gate must give the same answer for the same code, or engineers learn to rerun until green. The course's smoke tier runs seed 1 every time.

• Nightly and weekly runs use fresh seeds and record them. A nightly that reruns last night's seeds only re-proves last night. Fresh seeds find new failures; the recorded value brings any of them back.

• Promote seeds that matter. A seed that found a bug, or that alone hits a coverage bin, joins a version-controlled list that runs every night.

• Rerun-on-fail is a classifier, never a pass. Rerunning a failure with the same bundle tells you whether it is deterministic. Rerunning with a new seed and reporting green hides the bug. A run that fails then passes on an identical rerun is flaky and gets its own ticket.

What breaks: the crash that only happened sometimes

While collecting code coverage for this course, the coverage-instrumented build of the bench (the same bench built with Verilator's --coverage) ran dma_smoke_test with the same command, same seed and same binary, and died with a segmentation fault (exit code 139) in 8 of 20 runs. The plain build ran the same command 20 of 20 times cleanly. A crashed log has no UVM summary at all; it simply stops:

out_agt                axis_agent              -     @280
        mon                  axis_monitor            -     @682
          ap                 uvm_analysis_port       -     @690
      sb                     dma_scoreboard          -
$ echo $?
139

Nothing in the stimulus changed between a passing and a crashing run, so no amount of seed analysis could explain it: the cause is in the build or the tool, not in the DUT or the test. That is the definition of flaky, and the lesson is in how it was handled. The runs were not marked green on a retry; the crash was recorded as its own signature ("no summary"), kept out of the DUT bug count, and the coverage runs that were used were the ones that finished with a full summary. A regression that silently retried would have reported a clean night and hidden an unstable build.

Next: one failure replays. The nightly has thirty. Regression Triage by Signature decides how many bugs that is.

Related topics

• Up: Regression and Sign-off

• Seed Control and Regression Replay

• Custom plusargs