The interviewer describes a regression that fails once in fifty runs and asks how to capture it. The question tests method under uncertainty. The answer is a loop: reproduce, trigger, narrow.
Follow the arrows from the top. The two loops are where the time goes: each one repeats until its question answers yes.
The first loop ends only when the failure happens every time from the saved manifest.
The trigger sits once, between the loops, so every narrowing run is judged by the same checker.
The second loop ends when the smallest failing case fits one waveform window.
Reproduce first. Preserve the seed and random-stream setup, plusargs, source revision and local diff, tool version/options, defines, parameter overrides, input files, memory initialization, reset/clock schedule and external responses. Record hashes for replay inputs and scheduling/parallelism options. A seed alone is not a full run configuration.
Trigger second. Place a checker on the earliest observed divergence and retain pre-trigger history with a simulator-supported circular waveform buffer or an explicit transaction/state ring buffer. Freeze that buffer on failure and retain a bounded post-trigger tail. If only forward dumping is available, deterministic replay must enable dumping before the known failure window; enabling a dump at the failure cannot reconstruct earlier waves.
Narrow third. Vary one knob at a time, halve the stimulus, and walk the hierarchy from the top-level mismatch down to the block. Stop when the smallest failing case fits in one wave window.
Phase | Goal | Artifact kept |
|---|---|---|
Reproduce | The same failure on demand | Complete run manifest and immutable replay inputs |
Trigger | The earliest observed failing cycle | Checker plus buffered pre-trigger history or replay dump started before the window |
Narrow | The smallest failing case | Minimal stimulus plus hierarchy path |
The same seed passes in rerun. What does that mean? Keep several hypotheses open: omitted configuration/input differences, changed random-stream consumption, testbench driver/observer races, DUT scheduling races, uninitialized state or external nondeterminism. Compare manifests and input traces first; with identical inputs, find the first internal divergence and inspect event ordering. A bench race can expose a DUT race, so do not assume two independent bugs or assign blame from the seed alone.
The failure needs overnight runs to appear. How do you shorten the loop? Constrain the randomization toward the failing neighborhood and shorten the test while keeping the trigger armed. Each halving of runtime doubles the experiments per night.
“Rerun until it passes and move on.” A passing rerun without a root cause is not evidence. The bug ships with the design.
“Turn dumping on only after failure.” Forward dumping loses the causal history. Use a circular buffer, bounded state/transaction history, or deterministic replay with dumping enabled ahead of the known window; filter scope and time to keep it readable.
Add checks closer to the violated invariant and retain earlier state/transaction history. An earlier observation reduces the search, but still needs causal tracing; the trigger itself is not proof of root cause.
Change one at a time. With two moving knobs no result identifies the cause.