Hang and Deadlock Debug on Silicon

What this topic teaches

Hang and Deadlock Debug on Silicon converts bring-up know-how into staff-level execution decisions. Hangs look identical from the outside, but deadlock triage hinges on finding what stopped making forward progress first: CPU retirement, interconnect credits, DMA completion queues, or an always-on firmware state machine. Strong teams snapshot heartbeat counters and queue depths at fixed intervals, then align them with trigger-based trace capture around the final forward-progress event. One common war story is blaming software spin loops while the real issue is a circular wait across NoC virtual channels plus a rare low-power entry handshake; another is chasing fabric deadlock when the root cause is an interrupt storm starving a watchdog service thread. The debug pivot is to build a resource dependency graph from the captured state and prove at least one break condition for each cycle; if none exists, you have hard deadlock and need architectural relief, not just timeout tuning.

Senior-engineer framing question

When Mean time to identify first stuck resource and classify issue as hang, livelock, or true deadlock. regresses, can you isolate first failing boundary, prove mechanism with artifacts, assign owners, and close with rollback-safe validation?

SILICON BRING-UP FLOW - Hang and Deadlock Debug on Silicon

symptom intake and setup state freeze
      |
      v
dependency map: power/reset/clock/interface/firmware
      |
      v
instrumented experiment with one-variable branch
      |
      v
first failing boundary classification
      |
      v
bounded mitigation and replay validation
      |
      v
owner signoff with rollback criteria

Evidence to collect

• Primary metric: Mean time to identify first stuck resource and classify issue as hang, livelock, or true deadlock..

• Primary artifact: Forward-progress packet: heartbeat timeline, queue watermark dump, dependency graph, and deadlock/livelock classification note..

• Owners to include: NoC and fabric owner, firmware scheduler owner, power management owner, post-silicon debug owner, system software owner.

• One reproducible failing run and one matched comparator run.

• One fixed-metadata run with board, firmware, and corner tags locked.

Ownership layers

OWNERSHIP LAYERS - Hang and Deadlock Debug on Silicon

+----------------------+--------------------------------+--------------------------------+
| Team                 | Primary responsibility         | Closure artifact               |
+----------------------+--------------------------------+--------------------------------+
| NoC and fabric owner | hypothesis map and execution     | triage decision log            |
| firmware scheduler owner | stage behavior and software proof | boot/trace evidence packet     |
| power management owner | replay matrix and risk closure    | signoff memo + rollback gates  |
+----------------------+--------------------------------+--------------------------------+

Decision matrix

EVIDENCE MATRIX - Hang and Deadlock Debug on Silicon

+-------------------------------+--------------------------------+--------------------------------+-----------------------------+
| Evidence                      | Tells you                      | Does not prove                 | Next action                 |
+-------------------------------+--------------------------------+--------------------------------+-----------------------------+
| rail/current timeline         | sequencing and power health    | firmware or protocol integrity | align with stage logs       |
| stage checkpoint logs         | failing transition boundary    | electrical root cause          | correlate with scope traces |
| interface trace/decode        | protocol behavior and timing   | global platform readiness      | replay under fixed setup    |
| shmoo/corner matrix           | margin-sensitive fail region   | exact failing mechanism        | isolate with targeted tests |
| before/after replay packet    | mitigation movement quality    | long-run stability             | run soak and corner matrix  |
+-------------------------------+--------------------------------+--------------------------------+-----------------------------+

Key takeaways

• Classify first failing boundary before broad mitigation attempts.

• Tie each claim to one reproducible artifact and one owner action.

• Close with validation matrix plus rollback triggers for release safety.

Common pitfalls

• Changing many variables per run and losing causality.

• Treating intermittent failures as noise before preserving first-failure state.

• Declaring closure from one pass run without corner replay.

Silicon bring-up deep dive

Triage quality is measured by how quickly teams converge from symptom to proven root-cause class with minimal collateral churn.

Concept diagram

TRIAGE CONVERGENCE

symptom -> classify -> isolate -> prove -> bounded fix -> replay

Metric graph

TRIAGE EFFECTIVENESS

wide speculative edits   ██████
classified bounded fixes █████████

Metrics and artifacts to collect

• time-to-classification

• first-failure artifact completeness

• hypothesis branch conversion rate

• post-fix recurrence trend

Mini case study

Intermittent field-like failures closed faster once teams forced one-variable branch tests and owner-tagged evidence packets.

Debug branches

• Preserve first-failure state before reruns.

• Use disproof-oriented experiments to collapse cause tree quickly.

• Promote fixes only after recurrence tracking windows pass.

Senior review question

Ask: what is the first failing boundary, which artifact proves it, and who owns bounded closure?

Key takeaways

• Tie every bring-up claim to one reproducible setup state and one proving artifact.

• Prefer bounded fixes with clear owner and rollback trigger over broad multi-variable edits.

Common pitfalls

• Running parallel uncontrolled experiments and losing causality.

• Declaring closure without replaying across representative corners.

• Escalating severity before bench/setup hypotheses are disproven.