Silicon Bring-up · All levels

Hang and Deadlock Debug on Silicon: Debug Playbook

Debug Playbook for Hang and Deadlock Debug on Silicon.

Debug playbook

Debug Playbook for Hang and Deadlock Debug on Silicon is anchored on Mean time to identify first stuck resource and classify issue as hang, livelock, or true deadlock.. Convert observed behavior into mechanism-backed and owner-bound actions.

Freeze setup metadata and preserve first-failure state.
Locate first persistent boundary where behavior diverges.
Classify mechanism: dependency, margin, protocol, software, or silicon.
Apply one focused reproducer and one bounded fix.
Re-run replay, corner, and soak confidence matrix.

Review memo template

diagram

BRING-UP REVIEW MEMO - Failure Triage & Debug / Hang and Deadlock Debug on Silicon

1. Symptom
   - Failing metric: Mean time to identify first stuck resource and classify issue as hang, livelock, or true deadlock.
   - Trigger context: <board/firmware/corner/test window>
   - First failing boundary: <power/reset/clock/interface/firmware>

2. Mechanism hypothesis
   - Candidate mechanism: Hangs look identical from the outside, but deadlock triage hinges on finding what stopped making forward progress first: CPU retirement, interconnect credits, DMA completion queues, or an always-on firmware state machine. Strong teams snapshot heartbeat counters and queue depths at fixed intervals, then align them with trigger-based trace capture around the final forward-progress event. One common war story is blaming software spin loops while the real issue is a circular wait across NoC virtual channels plus a rare low-power entry handshake; another is chasing fabric deadlock when the root cause is an interrupt storm starving a watchdog service thread. The debug pivot is to build a resource dependency graph from the captured state and prove at least one break condition for each cycle; if none exists, you have hard deadlock and need architectural relief, not just timeout tuning.
   - Competing hypotheses: setup, dependency, margin, software path, silicon defect
   - Missing evidence: <trace/scope/register/report>

3. Proposed action
   - Smallest reversible change: <setup/script/config/firmware>
   - Expected movement: <repro rate/latency/pass trend>
   - Regression risk: stability, safety, release timeline, ownership handoff

4. Signoff
   - Required artifact: Forward-progress packet: heartbeat timeline, queue watermark dump, dependency graph, and deadlock/livelock classification note.
   - Required owners: NoC and fabric owner, firmware scheduler owner, power management owner, post-silicon debug owner, system software owner
   - Final decision: ship, bounded rollout, rollback, respin escalation

Silicon bring-up deep dive

Triage quality is measured by how quickly teams converge from symptom to proven root-cause class with minimal collateral churn.

Concept diagram

diagram

TRIAGE CONVERGENCE

symptom -> classify -> isolate -> prove -> bounded fix -> replay

Metric graph

diagram

TRIAGE EFFECTIVENESS

wide speculative edits   ██████
classified bounded fixes █████████

Metrics and artifacts to collect

time-to-classification
first-failure artifact completeness
hypothesis branch conversion rate
post-fix recurrence trend

Mini case study

Intermittent field-like failures closed faster once teams forced one-variable branch tests and owner-tagged evidence packets.

Debug branches

Preserve first-failure state before reruns.
Use disproof-oriented experiments to collapse cause tree quickly.
Promote fixes only after recurrence tracking windows pass.

Senior review question

Ask: what is the first failing boundary, which artifact proves it, and who owns bounded closure?

Key takeaways

Tie every bring-up claim to one reproducible setup state and one proving artifact.
Prefer bounded fixes with clear owner and rollback trigger over broad multi-variable edits.

Common pitfalls

Running parallel uncontrolled experiments and losing causality.
Declaring closure without replaying across representative corners.
Escalating severity before bench/setup hypotheses are disproven.

Debug ladder

Sequence: reproduce -> classify -> isolate -> instrument -> bounded fix -> replay.

Avoid parallel broad edits before first root-cause class is proven.

Practice this lesson