You push a one-line fix. A short set of checks runs and passes. That night, a longer run fails on your line. The short set never stood a chance against that bug, and the long run could never have finished before lunch. Both did their job, because they have different jobs.
Which checks run on every change, and which can wait until tonight?
The smoke test is small and runs on every push. Its job is to catch gross breakage quickly and block the change until it passes. In this project the smoke test is make all: lint with Verilator, simulation with the strong self-checking testbench, and a synthesis check with Yosys. Each step is deterministic: the same code gives the same verdict on every run.
The nightly regression is broad and runs on a schedule. A regression is a suite you re-run to catch new breakage in old behavior, and the nightly earns the name by doing the runs that are too slow or too wide for every push: many random seeds, the whole mutation matrix, coverage, and longer runs. Its job is confidence, and its price is time.
The figure below draws the two lanes side by side. Read the left lane first: it starts on every push and gates the change. Then read the right lane: it starts on a schedule and must be read in the morning.
Random stimulus draws a different input sequence for each seed, so one seed exercises one path through the design. A bug that needs a rare alignment of inputs stays hidden on most seeds and shows on a few. A single passing seed therefore says little: it may have missed the bug rather than refuted it.
The project keeps this honest in one line of the testbench: the random testbench prints its seed first, and every failure message names the seed and the cycle. A seed in the log turns a failure into a replay: run the same seed again and the same stimulus returns.
The run below drives fourteen 60-cycle random tests at a mutant whose counter sticks at 15 instead of wrapping to 0, measured on macOS with Icarus 13.0. Read the left column for the seed and the right column for the verdict: eleven seeds pass on broken code, and three fail:
# captured terminal output
$ iverilog -g2012 -s tb_random -o build/bug.vvp saturates.sv tb/tb_random.sv && for s in 1 2 3 4 5 6 7 8 9 10 11 12 13 14; do printf 'seed %-3s ' $s; vvp build/bug.vvp +seed=$s +cycles=60 | grep -E '^(PASS|FATAL)' | cut -c1-100; done
seed 1 PASS: seed=1, 61 cycles
seed 2 PASS: seed=2, 61 cycles
seed 3 PASS: seed=3, 61 cycles
seed 4 PASS: seed=4, 61 cycles
seed 5 PASS: seed=5, 61 cycles
seed 6 PASS: seed=6, 61 cycles
seed 7 FATAL: tb/tb_random.sv:41: FAIL: seed=7 cycle=43: rst_n=1 clear=0 valid=1 expected 0 got 15
seed 8 PASS: seed=8, 61 cycles
seed 9 PASS: seed=9, 61 cycles
seed 10 PASS: seed=10, 61 cycles
seed 11 PASS: seed=11, 61 cycles
seed 12 PASS: seed=12, 61 cycles
seed 13 FATAL: tb/tb_random.sv:41: FAIL: seed=13 cycle=25: rst_n=1 clear=0 valid=1 expected 0 got 15
seed 14 FATAL: tb/tb_random.sv:41: FAIL: seed=14 cycle=41: rst_n=1 clear=0 valid=1 expected 0 got 15The three failures agree exactly: with the counter at 15 and a valid event present, the design should wrap to 0 and instead holds 15. The eleven passes never lined up that precondition inside 60 cycles, so they prove nothing about the wrap. That is why the nightly lane runs many seeds: rare preconditions need many draws.
And the seed replays. Running seed 7 twice gives the same failure at the same cycle both times, measured on the same platform:
# captured terminal output
$ vvp build/bug.vvp +seed=7 +cycles=60 | grep -E '^(seed|FATAL)'; vvp build/bug.vvp +seed=7 +cycles=60 | grep -E '^(seed|FATAL)'
seed=7
FATAL: tb/tb_random.sv:41: FAIL: seed=7 cycle=43: rst_n=1 clear=0 valid=1 expected 0 got 15
seed=7
FATAL: tb/tb_random.sv:41: FAIL: seed=7 cycle=43: rst_n=1 clear=0 valid=1 expected 0 got 15Both outputs above are captured terminal output from real random-stimulus runs. The sweep covers seeds 1 to 14 at 60 cycles each against the saturating mutant, and the replay runs seed 7 twice to show the same failure twice.
The nightly lane holds every check that is too slow or too wide for a push: hundreds of random seeds across long runs, the full mutation matrix that replays every known bug shape against the current testbench, and the coverage report that says which lines and branches the tests never touched. Slow does not mean optional. It means scheduled.
One more check belongs there: the course repository's whole-project runner, scripts/tools/run-start-examples.mjs. It copies each example project into a fresh directory and runs the full verification from scratch: the tool-version table, make all with its PASS line and no ERROR or FAIL lines, the Python models, the assertion step under Verilator, and the entire mutation matrix compared against its expected results. That is the definition of a nightly check: too broad for a push, exactly right for a night.
Wrapping needs the count to climb all the way to 15 inside 60 cycles and then meet a valid event on exactly the next edge. Random stimulus rarely lines up both halves, so most seeds never present the exposing case. Short runs plus rare preconditions give a low hit rate, which is the nightly lane's whole argument.
Take the sweep's command and its first result line and stop there: seed 1 passes on the broken design, and a one-seed suite would call the mutant good. The output below is real, and it is the trap:
# captured terminal output
$ iverilog -g2012 -s tb_random -o build/bug.vvp saturates.sv tb/tb_random.sv && for s in 1 2 3 4 5 6 7 8 9 10 11 12 13 14; do printf 'seed %-3s ' $s; vvp build/bug.vvp +seed=$s +cycles=60 | grep -E '^(PASS|FATAL)' | cut -c1-100; done
seed 1 PASS: seed=1, 61 cyclesThe verdict is honest and useless: the seed never met the wrap, so it passed. The fix is the nightly habit: never judge a random test by one seed, run many, and keep every seed in the log so each failure replays.
Putting everything in the smoke lane. The suite grows until nobody waits for it, and then nobody runs it. Keep the smoke lane short enough that blocking on it stays painless, and move the rest to the night without apology.
Running random tests with no recorded seed. A failure without its seed is a story you cannot retell: the stimulus is gone and the bug may never show again. Print the seed before the run, as this project's testbench does, and treat a seedless failure log as a broken test.
Running a nightly lane nobody reads. A red night followed by green merges teaches the team to ignore it, and then it protects nothing. Give the nightly an owner, read it every morning, and let it block the release when it fails.
Rule: run on every change what you can afford to wait for, run overnight everything else, and read it in the morning.
Smoke lane: lint, the directed strong testbench, and the synthesis check. All three are fast and deterministic, so they gate every change without slowing it. Nightly lane: 500 random seeds, the mutation matrix, and the coverage report. All three need many runs or broad replay, so they buy confidence overnight instead of blocking the afternoon.
The same seed with the same cycle count, before touching any code. The log already names both, so the replay should reproduce the exact failure. Only a reproduced failure is safe to debug; anything else may be a different bug wearing the same message.
The lane, not the tests. A regression nobody reads is decoration. Name an owner, read it every morning, and make it block the release: red at night must mean stopped in the morning.
The smoke test runs on every change and gates it; the nightly runs everything slower and broader.
One random seed proves little: this bug showed on only 3 of 14 seeds.
A check nobody reads is decoration: give the nightly an owner and a gate.