"One of our random tests fails about once in 500 runs. Nobody can make it fail on demand, and the team wants to mark it flaky and move on. You own it now. What do you do?"
The interviewer has given you a number and a temptation. Which one do you question first?
What "500 runs" means. 500 different seeds, or one seed run 500 times? The first is a deterministic bug with a rare input. The second is real nondeterminism: a race, an uninitialised value, the environment (determinism and seeds).
A replay. Take one failing seed, same commit, same command, and run it again. If it fails every time, the repro already exists.
A measured rate with its uncertainty, by signature. Not "once in 500" but "3 in 500, all with this message", and what that number can and cannot mean (statistics of rare failures).
What the failing seeds share. A probe over every seed, failing and passing, looking for the condition only the failures have.
A higher rate, on purpose. Stress the state the bug needs (making the rate go up).
A proof sized in advance. How many clean runs, of what, rule out the old rate.
"Rerun it" ignores that a seeded test is already a repro: the failing seed is the command. "Look for the problem" is browsing. "Probably the simulator" is the it must be the tool trap, unchecked. And "mark it flaky" ships a bug that fails once in 500 runs here and once a day in a customer's system.
Report. "First: 500 seeds, or one seed 500 times? And do all the failures have the same message?"
Reproduce. "I replay a failing seed on the same commit. If it fails every time, the bug is deterministic and the seed is my repro. I also run it in a second simulator: if both fail the same way, it is the design or the bench, not the tool."
Measure. "I sweep a few hundred seeds and count failures by signature, with an interval. If the replay was not deterministic, that changes the whole plan: then I hunt the hidden input first."
Localise. "I probe every seed for candidate conditions and look for one that all the failing seeds have and the passing ones do not. That condition is my hypothesis."
Raise the rate. "I write a directed test, or bias the random one, towards that condition, so it fails every run instead of once in 500."
Fix and prove. "I fix the cause, rerun the old failing seeds and the directed test, and then sweep fresh seeds: enough that, if the old rate were still there, I would almost surely have seen it. I keep the directed test in the regression."
To get a rare failure with one cause, this run uses a scratch copy of dbglab with one of its bugs fixed (B4, the unreset FILL register) and the random test shortened to 20 operations with +OPS=20. One bug is left in it. Sweep 500 seeds, as the nightly would:
$ for s in $(seq 1 500); do tb/run.sh random $s icarus +OPS=20 | grep FAIL; done > fails.txt
$ wc -l < fails.txt
3
$ cat fails.txt
FAIL test=random seed=320 cycle=476: SCOREBOARD (final sweep): RAM word 253 (address 000013f4) holds 34f9eb8a, expected 97edaa4a (last written by DMA run #1)
FAIL test=random seed=451 cycle=375: SCOREBOARD: RAM word 749 (address 00001bb4) read 1de81775, expected b76ec824 (last written by DMA run #1)
FAIL test=random seed=487 cycle=719: SCOREBOARD (final sweep): RAM word 451 (address 0000170c) holds 09f065f7, expected 5e829a5a (last written by DMA run #1)Three in 500, and they look alike: a wrong but real value (not X), in a word last written by a DMA run. Treat them as one bug until something says otherwise.
$ tb/run.sh random 451 icarus +OPS=20
dbglab: test=random seed=451
FAIL test=random seed=451 cycle=375: SCOREBOARD: RAM word 749 (address 00001bb4) read 1de81775, expected b76ec824 (last written by DMA run #1)
$ tb/run.sh random 451 icarus +OPS=20
dbglab: test=random seed=451
FAIL test=random seed=451 cycle=375: SCOREBOARD: RAM word 749 (address 00001bb4) read 1de81775, expected b76ec824 (last written by DMA run #1)
$ tb/run.sh random 451 verilator +OPS=20
dbglab: test=random seed=451
FAIL test=random seed=451 cycle=375: SCOREBOARD: RAM word 749 (address 00001bb4) read 1de81775, expected b76ec824 (last written by DMA run #1)Same line, same cycle, twice and in a second simulator. Nothing about this test is flaky: seed 451 fails every time. "Once in 500" was a property of the stimulus, not of the run.
$ python3 rate.py 3 500 1500 # exact binomial interval; what 1500 clean runs would rule out3 in 500: rate 0.60%, 95% interval 0.12% to 1.74%
1500 clean runs: chance if the rate were still 0.60%: 1.2e-04; rate now below 0.20% (95%)
runs to see one failure at 0.60% with 95% confidence: 498"Once in 500" is consistent with these three failures, and so is once in 57. That width is why you do not plan from the point estimate. The last line answers a question the interviewer may ask: to see this bug even once with 95% confidence at its current rate takes about 500 runs, which is exactly why nobody could make it fail on demand by rerunning.
$ probes/run-probe.sh b4fix probes/dma_probe.sv random 451 +OPS=20
dbglab: test=random seed=451
cycle 25: DMA run #1 start LEN=298 SRC=000011f4 DST=00001934 left_q=42
cycle 363: DMA run #1 done words written=42 dst_q=000019dc
FAIL test=random seed=451 cycle=375: SCOREBOARD: RAM word 749 (address 00001bb4) read 1de81775, expected b76ec824 (last written by DMA run #1)
$ ./sweep-long.sh > sweep-long.txt # every seed: any DMA run over 255 words? did it fail?
$ awk '{print $2, $3}' sweep-long.txt | sort | uniq -c
497 long=0 fail=0
3 long=1 fail=1The DMA was asked for 298 words and wrote 42: 298 minus 256. Over all 500 seeds the pattern is exact: the three failing seeds are the only ones with a DMA run longer than 255 words. That is the hypothesis, an 8-bit beat counter, and the debug loop lesson walks this same bug to its line.
Stimulus | Failures | Mean run length | Failures per 1,000 cycles |
|---|---|---|---|
| 3 (0.6%) | 278 cycles | about 0.022 |
| 22 (22%) | 19,493 cycles | about 0.011 |
Running 70 times longer raised the rate 37 times, so per simulated cycle it found the bug half as often. Length is not the stress this bug needs; long DMA runs are. The debug loop lesson's directed dma_long test copies 255, 256, 257 and 511 words and fails at HEAD on every seed it was tried with (1, 2, 7 and 33). That is the stressed repro to prove the fix on.
$ cd ../b34fix # the same copy with the beat counter widened to 10 bits
$ for s in $(seq 1 1500); do tb/run.sh random $s icarus +OPS=20 | tail -1; done | cut -c1-4 | sort | uniq -c
1500 PASSAll 500 old seeds pass, including 320, 451 and 487, and so do 1,000 seeds that never ran before. If the old 0.60% rate were still there, 1,500 clean runs would happen with a chance of about one in 8,300. The claim is now a number: below 0.20% at 95% on this stimulus, plus a directed test that fails every seed without the fix.
The vague report became four facts: the failure is deterministic per seed, its rate is 0.12% to 1.74%, it needs a DMA run over 255 words, and the fix leaves no failure in 1,500 runs. Each answered a sentence of the strong answer, and none needed a waveform. The time went into sweeps that a script runs while you think.
"The same seed sometimes passes." Then something outside the seed differs: process order, an uninitialised value, a clock phase, the machine. Find it before anything else (why intermittents happen), for example by running the same seed with random start values or another simulator.
"The failure needs 2 million cycles before it shows." Shrink it: bisect the length, then the transactions (shrinking a failing test), and capture it with a flight recorder rather than a full dump (capturing the rare event).
"How many runs before you close it?" Decide the rate you must rule out and the confidence first. About 3/p clean runs rule out a rate p at 95%; at 0.6% that is 500, and a stressed repro needs far fewer.
"Two failure messages appeared in the sweep." Treat them as two bugs until one hypothesis explains both (keeping several hypotheses alive).
That the failure is not determined by the seed alone, so something else differs between the two runs: a different simulator or version, options, a plusarg, the environment, or a race or uninitialised value whose result changes between runs. Before anything else, compare the two runs' full records (version, options, plusargs, start values). In this lesson's copy, seed 451 failed identically three times, so that would point to a difference in how the colleague ran it.
It is a reasonable first check: at 22%, 30 clean runs would happen by luck about once in 1,700. But it only rules out a rate near 22% on that stimulus. It says little about a residual rate of 1%, and nothing about the short test where the bug was reported. Add the directed long-run test, the three old failing seeds, and a fresh sweep sized to the rate you want to rule out.
dbglab at HEAD 49c5f04 (build-history.sh), 2026-10-18, in scratch copies: b4fix is HEAD with B4's two-line FILL reset fix (one bug left, B3); b34fix adds B3's fix, localparam int CW = 10;. dbglab itself is unchanged. tb/run.sh random <s> icarus +OPS=20 over seeds 1 to 500 in b4fix: 3 FAIL (seeds 320, 451, 487), Icarus Verilog 13.0, 25 s. Seed 451 gives the same line twice in Icarus and once in Verilator 5.048.
dma_probe.sv (the debug-loop lesson's probe) compiled next to dbglab by probes/run-probe.sh on every seed: all 3 failing seeds contain a DMA run longer than 255 words, none of the 497 passing seeds does. Default length (+OPS unset) in b4fix, seeds 1 to 100: 22 FAIL; mean run length 278 cycles at +OPS=20 and 19,493 cycles at the default (seeds 1 to 100, b34fix). b34fix, +OPS=20, seeds 1 to 1500: 1500 PASS (64 s). Intervals from rate.py (Python 3, exact binomial).
Limits: one bug in simulation, and the 1-in-500 framing comes from shortening the random test, not from a real nightly. The bound holds for this bench's stimulus weights only. Interview wording is illustrative.