A random DCRouter test failed overnight. The engineer reruns it in the morning and it passes. The log shows the test name but not the seed; the regression script picked one at random and did not print it. The failure was real (lesson 4.9 showed the random hunt missing BUG_21 with four seeds out of twenty and catching it with the rest), and it is now gone. Without the seed, the only way back to it is luck.
A failure won't reproduce. What do you do?
The obvious response is to rerun until it fails again, or to close it as flaky. Rerunning with new seeds samples different tests (lesson 4.9); a rare failure may not return for hundreds of runs. Closing it discards a real bug. Reproducibility has to be designed in before the failure, and recovered methodically after.
Ingredient | Why it matters |
|---|---|
The seed | With random stability, the same seed and code give the same run (fact V7) |
The exact build | RTL, testbench, tools and versions; a testbench edit changes what a seed produces |
Plusargs and configuration | Test options, knobs, verbosity that alters timing or randomisation |
Environment | Simulator version, libraries, operating-system differences, licence-dependent options |
Non-determinism in the testbench | Races between processes, uninitialised variables, wall-clock time, hash ordering |
Same seed, same build? Check the log records both; rebuild the exact version from the commit if needed.
Different simulator, machine or options? Run where it failed, with the same options.
Race in the testbench? Look for processes writing the same variable at the same time, or blocking assignments to signals read by other processes at the same edge.
Uninitialised state? X versus 0 differences between simulators or options.
Truly rare? Run many seeds of the same test in parallel, with the failure signature as the stop condition (lesson 13.5).
Technique | Effect |
|---|---|
Start closer to the failure | Checkpoint or backdoor setup instead of replaying from reset (lessons 3.5, 10.6) |
Remove stimulus that does not matter | Delta debugging (lesson 13.4) |
Turn the case into a directed test | Survives testbench changes that break the seed |
Fix the seed in a regression list | The failure stays checked after the fix (lesson 4.9) |
Practice | Purpose |
|---|---|
Seed, build hash and command line printed in every log | Any failure can be rerun exactly |
Regression systems that store failing seeds and builds | Reproduction is one command |
Lint and race detection for testbench code | Non-determinism removed before it hides failures |
Failures never closed as flaky without root cause | Rare real bugs kept |
A team (illustrative) had a test that failed about once a week and was eventually marked flaky and excluded. Months later the same signature appeared in emulation. The cause was a real design race between a register write and a counter increment in the same cycle, which only some seeds produced. The excluded test had been reporting it all along.
Random stability ties values to the order of random calls; adding or moving a call changes what every seed produces.
The seed, the exact build (commit or hash of RTL and testbench), the command line and options, and the tool version.
A failure that cannot be reproduced cannot be debugged or its fix verified.
Record seed, build, options and environment for every run; remove testbench non-determinism.
Shrink reproductions into directed tests, and never close a failure as flaky without a cause.