A nightly regression reports one failure in 1,000 runs of a random test. The test passed the night before and passes when rerun with a new seed the next morning. Someone suggests it was a glitch and proposes closing the ticket. The seed of the failing run is in the log. Whether that one failure is noise or the most important result of the week depends entirely on what is done with that seed.
What does a seed mean, and how should random tests be run and trusted?
The obvious mental model treats a random test as one test that either passes or fails. It is really a family of tests, one per seed, each with different packets, timing and choices. A pass with one seed says that one member of the family passed. A rare failure says some members fail, and the seed identifies one of them exactly. Treating a random test as a single verdict throws both facts away.
Reproducible: with SystemVerilog random stability, the same seed and the same code reproduce the same run, cycle for cycle (fact V7).
Different seed, different test: each seed is an independent sample of the stimulus space.
Fragile across edits: adding or reordering a random call in the testbench changes what every seed produces, so a failing seed should be reproduced on the same build first.
If a bug is triggered and caught with probability r per seed, the chance that at least one of N seeds catches it is (fact M3):
A bug caught by one seed in a thousand (r = 0.001) has a 63% chance of being seen in 1,000 seeds and needs about 3,000 seeds for 95%.
Lesson 4.10's random hunt was built three times (clean, with BUG_20, with BUG_21) and each build was run with seeds 1 to 20 (Verilator's +verilator+seed+<n>). Each seed generates different traffic: between 194 and 354 packets in the five seeds sampled.
build seeds passing seeds catching the bug
clean 20 of 20 -
BUG_20 0 of 20 20 of 20
BUG_21 4 of 20 16 of 20 (missed with seeds 2, 5, 17, 18)How to read it: BUG_20 (sleeping with an output locked) is caught by every seed: the random rounds make mid-packet pauses common. BUG_21 (port 3 cannot wake the router) needs the router to fall asleep while input 3 is the only input with traffic left to send, for example when input 3 is alone in a round, or is the last slow source still sending; if any other input sends, it wakes the router for input 3. Four seeds never produce that situation. A single run of this test with seed 2 would pass with BUG_21 present. Two independent seeds would catch it with probability about 1 − 0.2² = 0.96.
Step | Why |
|---|---|
Rerun the failing seed on the same build | Confirms it reproduces; random stability makes this exact |
Debug it like any failure | It is a real behaviour of the design or the testbench, not a glitch |
Decide: design bug, testbench bug or illegal stimulus | All three happen; a rare seed often finds a rare legal case |
Turn it into a permanent test | Keep the seed in a list of fixed regression seeds, and write a directed test for the case, which survives testbench edits |
Estimate how rare the trigger is | One in 1,000 seeds says roughly how much coverage or bias is missing (lesson 4.5) |
Practice | Purpose |
|---|---|
Fresh random seeds every night, logged | Each regression explores new members of each test's family |
A fixed list of seeds that once failed | Fixed bugs stay fixed |
Many seeds of few tests, rather than one seed of many | Rare triggers need samples (the formula above) |
Coverage merged across all seeds | Shows what the family as a whole reached (Part 6) |
Never close an unreproduced failure | A missing seed or build is a process bug to fix |
A team (illustrative) ran each random test with one fixed seed for months, for "stable results". Their regression was in effect a directed suite whose cases they had never seen. Switching to fresh seeds nightly produced a burst of failures in the first week, every one a real bug that the fixed seeds had happened to avoid.
That seed never lets the router fall asleep while input 3 is the only input with traffic left, so the bug is never activated.
About 3,000, since (1 − 0.001)^3000 ≈ e^(−3) ≈ 0.05.
A random test is a family of tests; a seed names one member and reproduces it exactly.
A passing seed proves little; rare triggers need many seeds, as 1 − (1 − r)^N shows.
Log every seed, reproduce every failure on the same build, and keep failing seeds and directed tests for fixed bugs.