Intermittent and Marginal Failure Triage

What this topic teaches

Intermittent and Marginal Failure Triage converts bring-up know-how into staff-level execution decisions. Intermittent failures are where bring-up programs burn calendars: one in two thousand boots, only on two benches, only after thermal soak, and gone when heavy tracing is enabled. The practical playbook is statistical, not anecdotal. Engineers convert vague field descriptions into failure-rate curves by sweeping one stress axis at a time, including voltage margin, temperature ramps, memory traffic profile, and PLL spread, while preserving seed and instrumentation consistency. War-story teams learn to distrust first correlations; many apparent timing fixes were actually changing ISR load or DRAM training phase ordering. The goal is to move from ghost failures to parameterized windows where the bug is likely, then lock a high-probability repro harness that can validate mitigations without waiting days per experiment.

Senior-engineer framing question

When Reproducibility uplift and confidence interval for failure rate versus stress factor changes. regresses, can you isolate first failing boundary, prove mechanism with artifacts, assign owners, and close with rollback-safe validation?

SILICON BRING-UP FLOW - Intermittent and Marginal Failure Triage

symptom intake and setup state freeze
      |
      v
dependency map: power/reset/clock/interface/firmware
      |
      v
instrumented experiment with one-variable branch
      |
      v
first failing boundary classification
      |
      v
bounded mitigation and replay validation
      |
      v
owner signoff with rollback criteria

Evidence to collect

• Primary metric: Reproducibility uplift and confidence interval for failure rate versus stress factor changes..

• Primary artifact: Marginality dossier with shmoo-style failure map, reproducibility harness, and ranked environmental sensitivity table..

• Owners to include: silicon characterization owner, signal integrity owner, DRAM and PHY owner, firmware diagnostics owner, reliability engineering owner.

• One reproducible failing run and one matched comparator run.

• One fixed-metadata run with board, firmware, and corner tags locked.

Ownership layers

OWNERSHIP LAYERS - Intermittent and Marginal Failure Triage

+----------------------+--------------------------------+--------------------------------+
| Team                 | Primary responsibility         | Closure artifact               |
+----------------------+--------------------------------+--------------------------------+
| silicon characterization owner | hypothesis map and execution     | triage decision log            |
| signal integrity owner | stage behavior and software proof | boot/trace evidence packet     |
| DRAM and PHY owner | replay matrix and risk closure    | signoff memo + rollback gates  |
+----------------------+--------------------------------+--------------------------------+

Decision matrix

EVIDENCE MATRIX - Intermittent and Marginal Failure Triage

+-------------------------------+--------------------------------+--------------------------------+-----------------------------+
| Evidence                      | Tells you                      | Does not prove                 | Next action                 |
+-------------------------------+--------------------------------+--------------------------------+-----------------------------+
| rail/current timeline         | sequencing and power health    | firmware or protocol integrity | align with stage logs       |
| stage checkpoint logs         | failing transition boundary    | electrical root cause          | correlate with scope traces |
| interface trace/decode        | protocol behavior and timing   | global platform readiness      | replay under fixed setup    |
| shmoo/corner matrix           | margin-sensitive fail region   | exact failing mechanism        | isolate with targeted tests |
| before/after replay packet    | mitigation movement quality    | long-run stability             | run soak and corner matrix  |
+-------------------------------+--------------------------------+--------------------------------+-----------------------------+

Key takeaways

• Classify first failing boundary before broad mitigation attempts.

• Tie each claim to one reproducible artifact and one owner action.

• Close with validation matrix plus rollback triggers for release safety.

Common pitfalls

• Changing many variables per run and losing causality.

• Treating intermittent failures as noise before preserving first-failure state.

• Declaring closure from one pass run without corner replay.

Silicon bring-up deep dive

Triage quality is measured by how quickly teams converge from symptom to proven root-cause class with minimal collateral churn.

Concept diagram

TRIAGE CONVERGENCE

symptom -> classify -> isolate -> prove -> bounded fix -> replay

Metric graph

TRIAGE EFFECTIVENESS

wide speculative edits   ██████
classified bounded fixes █████████

Metrics and artifacts to collect

• time-to-classification

• first-failure artifact completeness

• hypothesis branch conversion rate

• post-fix recurrence trend

Mini case study

Intermittent field-like failures closed faster once teams forced one-variable branch tests and owner-tagged evidence packets.

Debug branches

• Preserve first-failure state before reruns.

• Use disproof-oriented experiments to collapse cause tree quickly.

• Promote fixes only after recurrence tracking windows pass.

Senior review question

Ask: what is the first failing boundary, which artifact proves it, and who owns bounded closure?

Key takeaways

• Tie every bring-up claim to one reproducible setup state and one proving artifact.

• Prefer bounded fixes with clear owner and rollback trigger over broad multi-variable edits.

Common pitfalls

• Running parallel uncontrolled experiments and losing causality.

• Declaring closure without replaying across representative corners.

• Escalating severity before bench/setup hypotheses are disproven.