Part 8 answers: It fails once a day. How do I catch it?
Intermittent failures defeat every method that needs a reliable repro, so the first job is to make them less intermittent. This part raises the failure rate, captures the rare event when it happens, and uses statistics to decide when a fix is really a fix.
Raise the failure rate of a rare bug by changing what stresses it.
Capture a rare event with triggers and a flight recorder.
Calculate how many clean runs prove a fix to a stated confidence.
7 lessons, about 73 min
The part on one page, taken from each lesson's Remember box.
Why Intermittents Happen
Hardware does not fail at random. Something differs between a passing run and a failing one; the job is to find what.
Unrelated clocks: the ratio sets the failure rate and the phase decides which events fail. Silicon picks a new phase at every power-up.
An unreset flop takes the value its simulator gives it: X, 0 or random. Only random start values behave like a chip.
Rule of thumb: before chasing an intermittent, write down which input you think differs between passing and failing runs, and how you will control it.
Making the Rate Go Up
Score a stress in failures per simulated millisecond (or per CPU hour), not per run: a longer run fails more often and costs just as much more.
Shape beats amount: the same 10% of stalled cycles in bursts gave 97 times the yield of single-cycle stalls.
At a 1:1 clock ratio the phase decides everything: 57% or 10% for the same setting. Sweep the phase, never just one ratio.
Rule of thumb: ask what state the bug needs, then stress that state, one knob at a time, measured in failures per unit of cost.
Capturing the Rare Event
A flight recorder costs little and is always on; the failing check freezes it. Make it deeper than the time from cause to detection.
Trigger on the earliest cheap sign you have. A precursor fires more often than the bug: count its false triggers and accept them.
Dump windows, not runs: four 40-cycle windows held both failures in 27 KB; the full dump of the same run was 55 MB.
Rule of thumb: never wait for a rare failure with nothing recording; arm a recorder and a trigger first, so the next occurrence is the last one you need.
Statistics of Rare Failures
To see a bug of rate p at least once with 95% confidence you need about 3/p runs: 29 for 10%, 299 for 1%, 2,995 for 0.1%.
After n clean runs the rate is below about 3/n at 95% (the rule of three). A clean sweep bounds the bug; it never proves zero.
Prove a fix on the stressed repro first: at 88% per run, 2 clean runs say as much as 29 at the default 10%.
Rule of thumb: state the confidence and the rate you are ruling out before you start the sweep, and size the sweep from them.
Ordering-Dependent Bugs
An ordering bug needs event B before event A, or both in one cycle. Random stimulus reaches that order only when latencies happen to line up.
Make each event's timing a knob and run every order: a small grid of directed runs beats a thousand random seeds.
The failing cells are the bug's specification. A fix must pass every cell, and the random test stays to find orders you did not list.
Rule of thumb: when a failure depends on timing, name the two events, make their order a knob, and run every order before you change the design.
Heisenbugs
If a failure disappears when you add observation, suspect the observation first: compare the quiet and the observed run before believing either.
Debug code must not take time from the driver, draw from the stimulus generator or read anything with a side effect.
A passive monitor watches the pins, samples at the edge and drives nothing. Check it: the FAIL line must be identical with it on and off.
Rule of thumb: observation must leave the failing line identical, cycle for cycle; check that before you trust what it shows.
An Intermittent on dbglab
Turn a rate into an order: the zero fill first fails 20 seeds in 20, the pattern fill first passes 20 in 20. That is the repro and the first half of the proof.
A precursor fires where the symptom cannot: the probe flagged the unwritten FILL in Verilator, which passes because its flops start at 0.
Prove on the repro, then on old and fresh seeds: 1,000 fresh clean seeds bound the rate below 0.30%. Making the bench pass is not a fix.
Rule of thumb: an intermittent is closed when the ordered repro, the old failing seeds and a sweep of fresh seeds all pass, and something in the regression would catch it again.
Practice question: Waveform debug: missing pulse in CDC due to narrow pulse width
Practice question: Pulse synchroniser: stretching pulses across clock domains
Practice question: D flip-flop with asynchronous reset
Code lab: CDC pulse stretch
Code lab: DFF async reset
Interview: run a mock interview on this topic