Routing and Flow Control — Debug Playbook

On-call / interview prompt

A queue keeps growing at two routers while neighboring links are underutilized. Walk the shortest path to root cause.

ARCHITECTURE ANALYSIS CHAIN

1. METRIC     — IPC, CPI, MPKI, bandwidth, latency, queue depth, stall cycles
2. HYPOTHESIS — microarch or system cause ordered by likelihood
3. EXPERIMENT — trace, PMU counter, simulation, or RTL probe
4. CHANGE      — pipeline, cache, NoC, or memory hierarchy adjustment
5. VALIDATION  — workload replay, regression suite, PPA impact

Reference workflow

1. Confirm workload, model tag, seed, and PMU/counter definitions
2. Open the primary report and capture worst metric
3. Correlate metric with workload phase, structure, traffic class, or RTL hierarchy
4. Apply smallest fix with documented hypothesis
5. Re-run required workload, PPA, and verification regressions

Mechanism to narrate

• Differentiate structural deadlock risk from transient congestion oscillation.

• Check if adaptive routing signal reacts too slowly for burst timescale.

• Inspect whether credit return path is contending with data traffic.

Common pitfalls

• Increasing every buffer blindly without checking contention source.

• Switching routing algorithm without revalidating ordering guarantees.

Staff-level debug discipline

For Routing and Flow Control, senior debug is branch-and-bound: reduce the search space quickly, keep experiments reversible, and avoid hiding a systematic issue behind one local fix.

Debug decision tree

1. Reproduce the failure with the same workload, model tag, seed, and counter setup.

2. Classify the failure as workload issue, model issue, microarchitecture issue, software issue, implementation issue, or true product limitation.

3. Run one cheap experiment that can falsify the leading hypothesis.

4. Prefer a fix that improves a cluster over one that only hides the worst line.

5. After the fix, re-check Routing/flow-control stress verification report and the likely regression surface: QoS guarantees, coherency response latency, and software-visible jitter..

Escalation triggers

• The failure crosses architecture, RTL, verification, software, PD, or product ownership.

• The proposed fix consumes area, power, latency, or verification margin needed elsewhere.

• The issue repeats across workloads or blocks, suggesting methodology or model root cause.

• The remaining risk is silicon-facing: Mis-tuned flow control can collapse effective bandwidth despite nominal link capacity..

Debug branch diagram

VISUAL MODEL — NoC and Interconnect Architecture / Routing and Flow Control

        workload / trace
              │
              ▼
   metric symptom (Routing/flow-control stress verification report)
              │
              ▼
     likely microarchitectural mechanism
              │
      ┌───────┼────────┐
      ▼       ▼        ▼
  pipeline  memory    fabric/coherency
  stalls    misses    queues / ordering
      │       │        │
      └───────┼────────┘
              ▼
        bounded design change
              │
              ▼
   validation workload + PPA regression

Tradeoff matrix

TRADEOFF MATRIX — Routing and Flow Control

+----------------------+----------------------+----------------------+----------------------+
| Option               | Helps                | Can hurt             | Validation needed    |
+----------------------+----------------------+----------------------+----------------------+
| Larger / wider block | peak perf, miss rate | area, power, timing  | workload sweep       |
| Smarter policy       | hit rate, QoS, IPC   | verification risk    | corner cases + PMU   |
| More buffering       | latency tails, stalls| deadlock, leakage    | stress traffic tests |
| Software contract    | locality, ordering   | portability, APIs    | production workload  |
+----------------------+----------------------+----------------------+----------------------+

Senior rule: pick the smallest change that proves or disproves the mechanism.

Architecture deep dive

NoC is a queueing system — bandwidth, latency, and deadlock are coupled.

Concept diagram

NoC TOPOLOGY SKETCH

CPU0 ──┐      ┌── LLC0 ── DRAM0
       R0 ─── R1
CPU1 ──┘      │
              R2 ─── R3 ── GPU/DMA
              │      │
             NPU    LLC1 ── DRAM1

Look for: hot links, cyclic dependencies, VC starvation, and tail latency.

Metric graph

LATENCY DISTRIBUTION

p50    ██████  32 ns
p90    ████████████  71 ns
p99    ████████████████████████  210 ns
p99.9  █████████████████████████████████  480 ns

Averages hide QoS failures.

Metrics and artifacts

• link utilization

• average latency by master

• retry/backpressure counts

• QoS violation log

Mini case study

Average latency looks fine but tail latency spikes for CPU coherent reads when GPU DMA runs. QoS and separate VCs fix the starvation without doubling link width.

Debug branches

• If deadlock, check credit loops and routing restrictions first.

• If latency tail long, inspect arbitration and buffer depth.

Senior review question

Ask: what single metric would prove this concept is working or failing on your workload?

Key takeaways

• Connect every architecture claim to a workload and measurable metric.

• State verification and PPA impact before proposing design changes.

Common pitfalls

• Feature-driven design without MPKI/IPC/bandwidth evidence.

• Ignoring coherency and NoC traffic in cache and accelerator sizing.

Study notes

Re-read this topic with one concrete workload.