Fails in the box, passes on the bench

The parts passed production test at 25 C and nominal settings. In the field, units in a hot enclosure report corrupted results a few times a day. In the lab one test, a memory copy with heavy switching, fails on a few units even at room temperature, and a firmware engineer finds that raising the supply by 50 mV makes everything pass.

Is it a design bug, a defective batch, or a design with no margin? And is the voltage bump a fix?

What people usually do, and why it fails

The RTL regression is rerun with the failing data and passes, because the logic is right. Then the failing units are called a bad batch and sent for failure analysis, which finds nothing, because nothing is broken. Then the voltage bump ships. It hides a setup problem at the cost of power and ageing, and if the real problem includes a hold path it makes that worse. Each failing condition gets its own bug report, and nobody sees that all of them are one picture.

The method: map, read, measure, then localise

1. Confirm it is marginal. Change one condition and watch the failure rate: voltage, frequency, temperature, the test. A rate that follows any of them is a margin question. A failure that follows none points elsewhere: Part 8 for intermittents.

2. Map it. Run the failing test over a grid of supply voltage and clock period, a shmoo, at more than one temperature, on more than one unit. The bring-up course covers how a lab builds and automates one.

3. Read the shape. A diagonal wall that moves to longer periods as voltage falls is setup-type: a path too slow. A band at high voltage or low temperature that fails at every period is hold-type: a path too fast for the clock skew. Holes and islands inside a passing region suggest noise, a resonance or a test that is itself marginal. Which bits fail and how the failure moves were covered in timing-only bugs.

4. Compare shmoos. Across tests: a wall that moves with the test is power integrity (switching-heavy tests droop the supply). Across temperature: the setup wall moves with temperature, usually towards longer periods when hot; at low voltage in recent processes, temperature inversion can make cold the slow corner, so map both. Across units: a spread is the process; a single outlier is a defect candidate.

5. Measure the margin. At the worst test and the worst corner, count the steps from the operating point to the wall. That number, not "it passes", is the result.

6. Then localise the path. Move to a stage that has timing in it: static timing at the failing corner, gate-level simulation with the delay file (gate-level simulation), or path-delay tests on the tester.

Do it: a timing-margin model swept like a shmoo

No chip, chamber or tester is available here, so this is a model, and everything below is labelled as one. Each "die" of margin_unit has two paths. A setup path: a register, 0.96 ns of logic at 0.90 V and 25 C, and the next clock edge. A hold path: one flop to the next through 0.15 ns of logic, where the capturing flop's clock arrives 0.105 ns late. Every delay scales with an alpha-power law, slower at low voltage and when hot, and the setup path's local supply droops when many bits switch. The bench builds 81 dies per condition, nine voltages by nine periods, at four conditions, and prints each shmoo the way silicon gives it: pass or fail, and nothing else.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

`timescale 1ns/1ps
// margin_unit: a timing model of two paths on one die at one supply voltage, temperature and
// clock period. It is a model, not a netlist. Delays follow an alpha-power law,
//   scale(V, T) = ((0.90 - 0.35) / (V - 0.35)) ** 1.3 * (1 + 0.0015 * (T - 25)),
// so every path is slower at low voltage and when hot.
//   setup path: register a, then 0.96 ns of logic (at 0.90 V, 25 C), captured by the next edge.
//     Switching makes the local supply droop: V - 0.05 V * (bits of a that toggle) / 32.
//   hold path: flop b1 to flop b2 through 0.15 ns of logic; b2's clock arrives 0.105 ns late
//     (clock skew, kept fixed here), so a path that gets too fast races through.
// NOISY = 0 counts a up (few bits toggle); NOISY = 1 flips almost every bit every cycle.
module margin_unit #(parameter real V = 0.9, T = 25.0, PERIOD = 1.0, parameter bit NOISY = 0) (
  output bit done, output int unsigned setup_errs, hold_errs
);
  localparam real SKEW = 0.105;
  function automatic real scale(input real v);
    return ((0.90 - 0.35) / (v - 0.35)) ** 1.3 * (1.0 + 0.0015 * (T - 25.0));
  endfunction
  function automatic logic [31:0] f(input logic [31:0] x); return x ^ {x[28:0], 3'b0}; endfunction

  logic clk = 1'b0, clk_late = 1'b0;
  initial begin #0.3; forever #(PERIOD / 2.0) clk = ~clk; end
  always @(clk) clk_late <= #(SKEW) clk;

  // Each launched value is kept with the time it reaches the capturing flop; a flop captures the
  // newest value that has arrived by its clock edge (a transport delay, the same in every simulator).
  logic [31:0] a = '0, pv [3], r = '0, want = '0;
  real         pt [3], bt [3];
  logic        b1 = 1'b0, bv [3], b2 = 1'b0, b2_exp = 1'b0;
  int unsigned n = 0;
  initial begin
    setup_errs = 0; hold_errs = 0; done = 1'b0;
    for (int k = 0; k < 3; k++) begin pv[k] = '0; pt[k] = -10.0; bv[k] = 1'b0; bt[k] = -10.0; end
  end

  function automatic logic [31:0] arrived_p(input real now);
    for (int k = 0; k < 3; k++) if (pt[k] <= now) return pv[k];
    return 'x;
  endfunction
  function automatic logic arrived_b(input real now);
    for (int k = 0; k < 3; k++) if (bt[k] <= now) return bv[k];
    return 1'bx;
  endfunction

  always @(posedge clk) begin : launch_and_capture
    logic [31:0] nxt;
    real d;
    r = arrived_p($realtime);                      // setup path: capture at this edge
    if (n > 4 && n < 200) begin
      if (r !== want)    setup_errs++;
      if (b2 !== b2_exp) hold_errs++;
    end
    nxt = NOISY ? ~a ^ 32'(n) : a + 32'd1;         // launch the next value
    d = 0.96 * scale(V - 0.05 * $countones(nxt ^ a) / 32.0);
    for (int k = 2; k > 0; k--) begin pv[k] = pv[k-1]; pt[k] = pt[k-1]; bv[k] = bv[k-1]; bt[k] = bt[k-1]; end
    pv[0] = f(nxt); pt[0] = $realtime + d;
    bv[0] = ~b1;    bt[0] = $realtime + 0.15 * scale(V);
    b2_exp = b1;                                   // b2 must take b1's value from before this edge
    want = f(nxt);                                 // the next edge must capture this launch
    a = nxt; b1 = ~b1;
    n++;
    if (n == 200) done = 1'b1;
  end
  always @(posedge clk_late) b2 = arrived_b($realtime);   // hold path: capture 0.105 ns late
endmodule

The shmoo at room temperature

$ iverilog -g2012 -o s.vvp shmoo_tb.sv
$ vvp -n s.vvp
shmoo 0: 25 C, counting pattern   (* pass, . fail)
  period ns  0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7
  1.10 V     .   .   .   .   .   .   .   .   .
  1.05 V     *   *   *   *   *   *   *   *   *
  1.00 V     *   *   *   *   *   *   *   *   *
  0.95 V     *   *   *   *   *   *   *   *   *
  0.90 V     .   *   *   *   *   *   *   *   *
  0.85 V     .   .   .   *   *   *   *   *   *
  0.80 V     .   .   .   .   *   *   *   *   *
  0.75 V     .   .   .   .   .   .   .   *   *
  0.70 V     .   .   .   .   .   .   .   .   .
...

Read it as the lab would, with no knowledge of the paths. Two failure regions, two shapes:

• The lower left is a wall. As the voltage falls the first passing period moves right: 0.9 ns at 0.95 V, 1.2 ns at 0.85 V, 1.6 ns at 0.75 V. A failure that a longer period cures is a path that is too slow: setup-type.

• The top row is a band. At 1.10 V the part fails at every period, including 1.7 ns, nearly twice the shortest period that passes one row below. A slower clock does not help, so the period is not in the failing check: hold-type, a path that got fast enough to race the clock skew.

• The operating point (0.90 V, 1.0 ns) passes, with one step to the wall: 0.9 ns fails.

The picture

The colours come from +WHY, which only a model can print: grey is the setup path, accent the hold path. On silicon you get the stars and dots, and the shape is how you tell them apart.

Change the test, then the temperature

$ vvp -n s.vvp
...
shmoo 1: 25 C, switching pattern   (* pass, . fail)
  period ns  0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7
  1.10 V     .   .   .   .   .   .   .   .   .
  1.05 V     *   *   *   *   *   *   *   *   *
  1.00 V     *   *   *   *   *   *   *   *   *
  0.95 V     .   *   *   *   *   *   *   *   *
  0.90 V     .   .   *   *   *   *   *   *   *
  0.85 V     .   .   .   .   *   *   *   *   *
  0.80 V     .   .   .   .   .   .   *   *   *
  0.75 V     .   .   .   .   .   .   .   .   .
  0.70 V     .   .   .   .   .   .   .   .   .
shmoo 2: 85 C, counting pattern   (* pass, . fail)
  period ns  0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7
  1.10 V     *   *   *   *   *   *   *   *   *
  1.05 V     *   *   *   *   *   *   *   *   *
  1.00 V     *   *   *   *   *   *   *   *   *
  0.95 V     .   *   *   *   *   *   *   *   *
  0.90 V     .   .   *   *   *   *   *   *   *
  0.85 V     .   .   .   .   *   *   *   *   *
  0.80 V     .   .   .   .   .   .   *   *   *
  0.75 V     .   .   .   .   .   .   .   .   *
  0.70 V     .   .   .   .   .   .   .   .   .
shmoo 3: -40 C, counting pattern   (* pass, . fail)
  period ns  0.9 1.0 1.1 1.2 1.3 1.4 1.5 1.6 1.7
  1.10 V     .   .   .   .   .   .   .   .   .
  1.05 V     .   .   .   .   .   .   .   .   .
  1.00 V     *   *   *   *   *   *   *   *   *
  0.95 V     *   *   *   *   *   *   *   *   *
  0.90 V     *   *   *   *   *   *   *   *   *
  0.85 V     .   .   *   *   *   *   *   *   *
  0.80 V     .   .   .   *   *   *   *   *   *
  0.75 V     .   .   .   .   .   .   .   *   *
  0.70 V     .   .   .   .   .   .   .   .   *
PASS: setup wall moves with voltage, temperature and pattern; the hold band ignores the clock period

• Shmoo 1, same die, a switching-heavy test. The wall moves about one voltage step: at 0.90 V the first passing period is now 1.1 ns, so the operating point fails. Nothing about the logic changed; the supply droops when most bits switch. This is the scene's memory-copy test, and why one test fails at room temperature while others pass.

• Shmoo 2, hot. The wall moves the same way, and the operating point fails again: the field failure. The hold band has gone, because hot paths are slow.

• Shmoo 3, cold. The setup wall moves left (cold is fast in this model), and the hold band grows to two rows: 1.05 V now fails at every period. The voltage bump from the scene, 0.90 to 0.95 V, still passes cold; a second bump to 1.05 V would fail every cold unit.

• Verilator 5.048 prints the same four grids, and the same +WHY labels.

What the evidence says

One picture explains all three reports in the scene: there is no bad batch, the design has no setup margin at its worst test or its hot corner, and the voltage bump works for setup only because it stays below the hold band. The next stage is the one that has the paths: static timing at the hot, low-voltage corner with the switching test's droop, and gate-level simulation with the delay file on the paths it names. Only then do you choose between a slower clock, a higher voltage with its cost, and a fix to the path.

Traps

Comparing a bench and a tester

A shmoo from a bench board and one from a production tester can differ because the two set voltage at different points, measure temperature differently, or apply different patterns. Before reading a difference between them as a silicon effect, correlate the setups (ATE vs bench correlation). The same rule, that inputs must match before outputs are compared, is the subject of the correlation lesson.

Related topics

• Part 6 · Debug by Stage

• Bring-up: building and reading shmoo plots

• Bring-up: timing and voltage margin analysis

• Bring-up: intermittent and marginal failure triage

• Timing-only bugs