Computer Architecture · All levels

NoC Debug and Observability — Worked Example

Worked Example for NoC Debug and Observability (NoC and Interconnect Architecture).

Scenario

A customer workload triggers sporadic frame drops; trace correlation reveals priority inversion in one arbitration domain.

Timeline

Metric fails at review meeting
Engineer captures metric report, trace snippet, and workload phase
Root cause traced to incorrect assumption from prior stage
Minimal architecture change or policy experiment applied and documented
Workload regression matrix re-run on the tagged model

Outcome

Age-based arbitration patch plus watchdog alert threshold update removes drops and improves triage confidence.

Senior debrief

After solving the example, write the debrief a lead would expect: what changed, why it worked, what could regress, and what permanent methodology update prevents recurrence.

diagram

STAFF ARCHITECTURE REVIEW MEMO — NoC and Interconnect Architecture / NoC Debug and Observability

1. Current state
   - Failing / watched metric: NoC observability readiness report
   - Workload / benchmark / trace: <fill before review>
   - Model tag, RTL tag, simulator version, PMU setup: <fill before review>
   - Scope: core, cache level, NoC path, coherency domain, accelerator, or SoC budget

2. Root-cause hypothesis
   - Most likely mechanism: <name pipeline/cache/NoC/coherency/perf mechanism>
   - Competing hypothesis: <name the second plausible cause>
   - Evidence still missing: <counter, trace, waveform, model sweep, or workload slice>

3. Proposed action
   - Minimal reversible change: <microarchitecture, policy, sizing, traffic, or software contract change>
   - Expected improvement: <metric delta>
   - Regression risk: Weak observability turns recoverable performance bugs into schedule-threatening mystery failures.

4. Regression and signoff
   - Re-run: NoC observability readiness report
   - Must not regress: Silicon bring-up velocity, customer issue turnaround, and long-term platform reliability.
   - Decision owner: Silicon debug lead

Before / after metric graph

diagram

METRIC TREND GRAPH — NoC Debug and Observability

IPC / throughput
  ^
  |                  target
  |                 ─ ─ ─ ─ ─ ─ ─
  |            ● after bounded fix
  |         /
  |    ● baseline
  |  /
  |● failing run
  +---------------------------------> experiment index
    bad tag       hypothesis        accepted fix

Readout rule:
  - one dot is not a conclusion
  - compare against same workload, seed, model tag, and counter setup
  - explain why the fix moved the metric, not just that it moved

Architecture deep dive

NoC is a queueing system — bandwidth, latency, and deadlock are coupled.

Concept diagram

diagram

NoC TOPOLOGY SKETCH

CPU0 ──┐      ┌── LLC0 ── DRAM0
       R0 ─── R1
CPU1 ──┘      │
              R2 ─── R3 ── GPU/DMA
              │      │
             NPU    LLC1 ── DRAM1

Look for: hot links, cyclic dependencies, VC starvation, and tail latency.

Metric graph

diagram

LATENCY DISTRIBUTION

p50    ██████  32 ns
p90    ████████████  71 ns
p99    ████████████████████████  210 ns
p99.9  █████████████████████████████████  480 ns

Averages hide QoS failures.

Metrics and artifacts

link utilization
average latency by master
retry/backpressure counts
QoS violation log

Mini case study

Average latency looks fine but tail latency spikes for CPU coherent reads when GPU DMA runs. QoS and separate VCs fix the starvation without doubling link width.

Debug branches

If deadlock, check credit loops and routing restrictions first.
If latency tail long, inspect arbitration and buffer depth.

Senior review question

Ask: what single metric would prove this concept is working or failing on your workload?

Key takeaways

Connect every architecture claim to a workload and measurable metric.
State verification and PPA impact before proposing design changes.

Common pitfalls

Feature-driven design without MPKI/IPC/bandwidth evidence.
Ignoring coherency and NoC traffic in cache and accelerator sizing.

Study notes

Re-read this topic with one concrete workload.

Practice this lesson