Silicon Bring-up · All levels

Intermittent and Marginal Failure Triage: Debug Playbook

Debug Playbook for Intermittent and Marginal Failure Triage.

Debug playbook

Debug Playbook for Intermittent and Marginal Failure Triage is anchored on Reproducibility uplift and confidence interval for failure rate versus stress factor changes.. Convert observed behavior into mechanism-backed and owner-bound actions.

Freeze setup metadata and preserve first-failure state.
Locate first persistent boundary where behavior diverges.
Classify mechanism: dependency, margin, protocol, software, or silicon.
Apply one focused reproducer and one bounded fix.
Re-run replay, corner, and soak confidence matrix.

Review memo template

diagram

BRING-UP REVIEW MEMO - Failure Triage & Debug / Intermittent and Marginal Failure Triage

1. Symptom
   - Failing metric: Reproducibility uplift and confidence interval for failure rate versus stress factor changes.
   - Trigger context: <board/firmware/corner/test window>
   - First failing boundary: <power/reset/clock/interface/firmware>

2. Mechanism hypothesis
   - Candidate mechanism: Intermittent failures are where bring-up programs burn calendars: one in two thousand boots, only on two benches, only after thermal soak, and gone when heavy tracing is enabled. The practical playbook is statistical, not anecdotal. Engineers convert vague field descriptions into failure-rate curves by sweeping one stress axis at a time, including voltage margin, temperature ramps, memory traffic profile, and PLL spread, while preserving seed and instrumentation consistency. War-story teams learn to distrust first correlations; many apparent timing fixes were actually changing ISR load or DRAM training phase ordering. The goal is to move from ghost failures to parameterized windows where the bug is likely, then lock a high-probability repro harness that can validate mitigations without waiting days per experiment.
   - Competing hypotheses: setup, dependency, margin, software path, silicon defect
   - Missing evidence: <trace/scope/register/report>

3. Proposed action
   - Smallest reversible change: <setup/script/config/firmware>
   - Expected movement: <repro rate/latency/pass trend>
   - Regression risk: stability, safety, release timeline, ownership handoff

4. Signoff
   - Required artifact: Marginality dossier with shmoo-style failure map, reproducibility harness, and ranked environmental sensitivity table.
   - Required owners: silicon characterization owner, signal integrity owner, DRAM and PHY owner, firmware diagnostics owner, reliability engineering owner
   - Final decision: ship, bounded rollout, rollback, respin escalation

Silicon bring-up deep dive

Triage quality is measured by how quickly teams converge from symptom to proven root-cause class with minimal collateral churn.

Concept diagram

diagram

TRIAGE CONVERGENCE

symptom -> classify -> isolate -> prove -> bounded fix -> replay

Metric graph

diagram

TRIAGE EFFECTIVENESS

wide speculative edits   ██████
classified bounded fixes █████████

Metrics and artifacts to collect

time-to-classification
first-failure artifact completeness
hypothesis branch conversion rate
post-fix recurrence trend

Mini case study

Intermittent field-like failures closed faster once teams forced one-variable branch tests and owner-tagged evidence packets.

Debug branches

Preserve first-failure state before reruns.
Use disproof-oriented experiments to collapse cause tree quickly.
Promote fixes only after recurrence tracking windows pass.

Senior review question

Ask: what is the first failing boundary, which artifact proves it, and who owns bounded closure?

Key takeaways

Tie every bring-up claim to one reproducible setup state and one proving artifact.
Prefer bounded fixes with clear owner and rollback trigger over broad multi-variable edits.

Common pitfalls

Running parallel uncontrolled experiments and losing causality.
Declaring closure without replaying across representative corners.
Escalating severity before bench/setup hypotheses are disproven.

Debug ladder

Sequence: reproduce -> classify -> isolate -> instrument -> bounded fix -> replay.

Avoid parallel broad edits before first root-cause class is proven.

Practice this lesson