Ninety-nine green nights and one red one

A block-level regression has run the same 400 tests every night for three months. Last Tuesday one test failed: an event counter in a slow peripheral was one short. Wednesday it passed again, with the same seed, the same commit and the same simulator. In the lab the same counter is one short about once an hour, and never on the same event twice.

Everyone calls it random. It is not: something was different on Tuesday. What could it be, and how do you find out which?

What people usually do, and why it fails

The usual moves are to rerun until it passes, mark the test as flaky, or wait for it to happen again with more logging turned on. Each treats the failure as noise. Rerunning tells you nothing if the input that matters is not the one you control. Waiting is slow, and as a later lesson shows, extra logging can change the very thing you are waiting for.

The deeper mistake is to treat every intermittent as one kind of problem. A rare stimulus path, two events that collide, two clocks that drift against each other and a flop that powers up wrong all look the same in a pass-fail summary. They need different experiments.

The method: find the hidden input

An intermittent failure is a deterministic failure of an input you are not controlling. Logic computes the same answer from the same inputs every time. So list the inputs that can differ between two runs, and find the one that does. Five cover almost every case:

1. Prove the logic is deterministic in your run. Rerun the same seed, commit and command. If it always fails, the hidden input is the stimulus: go to Part 1's method.

2. If the same command sometimes passes, list what was not pinned: start values, clock phases, the simulator and its options, files from a previous run, the machine.

3. Read the rate. A rate that moves with stimulus weights points at a path; one set by two frequencies points at a crossing; one near 50% with a single unreset bit points at a start value.

4. Make the suspect input explicit. Turn it into a plusarg or a parameter, so a run that failed can be repeated exactly.

5. Sweep it. A curve that matches a prediction tells you the mechanism before you open a waveform.

The picture: a slow clock steps over a short pulse

Every 13 ns edge samples whatever is on the wire at that instant. A pulse 10 ns wide contains an edge only if one happens to fall inside it, so the chance of losing it is the share of time not covered: 1 − 10/13, about 23%. The ratio sets the rate. The phase, where the first edge falls, decides which pulses are lost.

Do it: two hidden inputs, made explicit

Clock ratio and phase

The design below takes a one-cycle event from a 10 ns domain into a slower one through two flops. The bench sends 1,000 events at seeded random times and counts how many the slow side saw. The fix, a toggle crossing, turns each event into a change of level, which no edge can step over. The destination period and its phase are plusargs, so both inputs are explicit.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

// pulse_to_slow: a one-cycle event in the src domain, seen in the dst domain through two flops.
// The bug: the pulse is one src cycle wide, so a slower dst clock can step over it.
module pulse_to_slow (
  input  logic dst_clk, rst_n, ev,
  output logic seen                           // one dst-cycle pulse per event that got through
);
  logic s1, s2, s3;
  always_ff @(posedge dst_clk or negedge rst_n)
    if (!rst_n) {s1, s2, s3} <= '0;
    else        {s1, s2, s3} <= {ev, s1, s2};
  assign seen = s2 && !s3;
endmodule

// The fix: the src side turns each event into a toggle of a level, which dst cannot step over.
module toggle_to_slow (
  input  logic src_clk, dst_clk, rst_n, ev,
  output logic seen
);
  logic t, s1, s2, s3;
  always_ff @(posedge src_clk or negedge rst_n)
    if (!rst_n) t <= 1'b0; else if (ev) t <= ~t;
  always_ff @(posedge dst_clk or negedge rst_n)
    if (!rst_n) {s1, s2, s3} <= '0;
    else        {s1, s2, s3} <= {t, s1, s2};
  assign seen = s2 ^ s3;
endmodule

// ratio_tb: the src clock is 10 ns. Each set sends 1000 events, 8 to 23 src cycles apart (seeded),
// and counts the events each crossing lost. +DST=<ps> runs one dst period and +PHASE=<ps> delays its first edge; with no plusargs it sweeps the dst period and checks the lesson's claims.
`timescale 1ns/1ps

$ iverilog -g2012 -o r.vvp ratio_tb.sv && vvp -n r.vvp
dst 10000 ps: pulse lost   0 of 1000 ( 0.0%, predicted  0.0%), toggle lost 0
dst 10100 ps: pulse lost  11 of 1000 ( 1.1%, predicted  1.0%), toggle lost 0
dst 11000 ps: pulse lost  95 of 1000 ( 9.5%, predicted  9.1%), toggle lost 0
dst 13000 ps: pulse lost 241 of 1000 (24.1%, predicted 23.1%), toggle lost 0
dst 16000 ps: pulse lost 381 of 1000 (38.1%, predicted 37.5%), toggle lost 0
dst 20000 ps: pulse lost 498 of 1000 (49.8%, predicted 50.0%), toggle lost 0
PASS: the loss rate follows 1 - Tsrc/Tdst, and the toggle crossing loses nothing

Read the rows as a curve. With equal clocks nothing is lost. A destination only 1% slower loses 1.1% of events: a failure once in a hundred, the kind that survives months of regressions at one fixed ratio. Every row lands within a point or so of the prediction 1 − Tsrc/Tdst, which is the evidence that the mechanism is the one in the picture. Now hold the ratio and move only the phase:

$ for p in 0 3000 6000 9000; do vvp -n r.vvp +DST=13000 +PHASE=$p; done
dst 13000 ps, phase 0 ps: pulse lost 208 of 1000 (events 1 7 12 15 18 23 ...), toggle lost 0
dst 13000 ps, phase 3000 ps: pulse lost 236 of 1000 (events 0 9 10 14 37 39 ...), toggle lost 0
dst 13000 ps, phase 6000 ps: pulse lost 237 of 1000 (events 3 5 6 8 20 22 ...), toggle lost 0
dst 13000 ps, phase 9000 ps: pulse lost 246 of 1000 (events 2 11 13 16 19 21 ...), toggle lost 0

Same stimulus, same ratio, the same rate within sampling noise, and a completely different list of lost events. A chip picks a new phase every time it powers up, so on the bench the lost event moves from run to run and looks random. In simulation the phase is one fixed number unless you make it a knob. Verilator printed identical lines for both commands.

A start value

The second design joins pairs of 16-bit halves into words. Its half-select flop ph has no reset, on the argument that it is "just a toggle". Whether it works depends on the value ph powers up with, which no seed controls. The default bench forces both possible values and checks both designs; +ONE leaves ph to the simulator.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

// pack2: joins two 16-bit halves into one 32-bit word, low half first.
// The bug: the half-select flop ph has no reset, so the first half lands wherever ph powered up.
module pack2 (
  input  logic        clk, rst_n, in_valid,
  input  logic [15:0] in_half,
  output logic        out_valid,
  output logic [31:0] out_word
);
  logic        ph;                            // 0: expecting the low half
  logic [15:0] lo_q;
  always_ff @(posedge clk)                    // no reset: "it is just a toggle"
    if (in_valid) ph <= ~ph;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin out_valid <= 1'b0; lo_q <= '0; out_word <= '0; end
    else begin
      out_valid <= in_valid && ph;
      if (in_valid && !ph) lo_q <= in_half;
      if (in_valid &&  ph) out_word <= {in_half, lo_q};
    end
endmodule

// The fix: ph is reset with everything else.
module pack2_fixed (
  input  logic        clk, rst_n, in_valid,
  input  logic [15:0] in_half,
  output logic        out_valid,
  output logic [31:0] out_word
);
  logic        ph;
  logic [15:0] lo_q;
  always_ff @(posedge clk or negedge rst_n)
    if (!rst_n) begin ph <= 1'b0; out_valid <= 1'b0; lo_q <= '0; out_word <= '0; end
    else begin
      if (in_valid) ph <= ~ph;
      out_valid <= in_valid && ph;
      if (in_valid && !ph) lo_q <= in_half;
      if (in_valid &&  ph) out_word <= {in_half, lo_q};
    end
endmodule

$ iverilog -g2012 -o i.vvp init_tb.sv && vvp -n i.vvp
power-up ph=0: pack2 0 of 8 wrong, pack2_fixed 0 wrong
power-up ph=1: pack2 8 of 8 wrong, pack2_fixed 0 wrong
PASS: pack2 works or fails by its power-up state; pack2_fixed works from both
$ vvp -n i.vvp +ONE
FAIL: pack2 started with ph=x: 0 words, 0 wrong
$ vi/Vinit_tb +ONE                          # Verilator, default build
PASS: pack2 started with ph=0, 8 words right

Three simulations, three answers. Icarus starts ph at X, and an X toggled stays X, so no word ever comes out: a loud failure. Verilator's default starts it at 0, which happens to be the good value, so the bug is invisible. A chip starts it at whatever the flop settles to. Verilator can model that:

1
2
3
4
5
6

$ verilator --binary --timing --assert -Wno-fatal --x-assign unique --x-initial unique \
    --top-module init_tb init_tb.sv -Mdir vx
$ for s in $(seq 1 20); do vx/Vinit_tb +ONE +verilator+rand+reset+2 +verilator+seed+$s \
    | grep -E "^(PASS|FAIL)"; done | sort | uniq -c
   8 FAIL: pack2 started with ph=1: 8 words, 8 wrong
  12 PASS: pack2 started with ph=0, 8 words right

Eight failures in twenty with nothing changed but the start values. That is the bench equivalent of "fails after some power cycles". The +verilator+seed argument makes each start value repeatable, which turns the intermittent back into a failing command: seed 2 fails every time.

What the evidence says

Both designs are deterministic. Each failure came from an input that a normal run does not vary: the phase between two clocks, or the value of one flop at time zero. Once that input is a plusarg, the failure repeats on demand, the rate can be predicted, and the fix can be checked against every value of the input, not just the lucky one. The toggle crossing lost nothing at any ratio; the reset ph passed from both start values.

You are on the Reproduce step of the debug loop: the output is a command that fails every time, for a reason you can name. The rest of this part builds on that: raising a rate on purpose, capturing an event you cannot repeat, and deciding how many clean runs prove a fix.

Traps

Related topics

• Part 8 · Intermittent Failures

• Next: making the rate go up

• CDC bugs: the class

• CDC/RDC: debug and case studies

• Bring-up: failure triage in the lab