Every NoC symptom falls into one of four quadrants. Classify before touching knobs, because each quadrant has a different owner and a different fix.
Quadrant | Counters say | Likely cause |
|---|---|---|
Injection burst | Source queues deep, fabric idle | Synchronized senders exceed the plan |
Arbitration delay | Grant age skews to one class | Unfair policy or wrong weights |
Credit stall | Local queues grow, returns slow, neighbors idle | Shallow pool or stuck return path |
Endpoint backpressure | Destination accept rate sags | Slow sink, not a fabric fault |
Compare offered load against achieved throughput per region, with a fixed deterministic seed. A worked hotspot example with stated assumptions: four DMA flows at 0.25 flits per cycle share one XY corridor link.
Assumed offered load: 4 DMA flows * 0.25 flits per cycle = 1.0 flits per cycle on one link
link capacity 1 flit per cycle -> utilization 1.0, saturated; tail latency climbs while the mean lags
remap two flows to a parallel corridor: 2 * 0.25 = 0.5 per link -> utilization 0.5, queues drain
assumed traffic mix for the scenario: CPU 30% + coherent refill 25% + DMA 35% + IO 10% = 100%Reproduce with the same seed, overlay hot links on the floorplan, apply exactly one intervention, then recheck p99 and starvation counters. Two changes at once teach nothing.
Trigger on precursors, not on the spike: queue-slope, grant-age, and credit-latency thresholds open a capture window around the event. Timestamp alignment across clock domains is mandatory, with a stated error budget.
Watchdog: link round trip 12 cycles -> timeout 4 * 12 = 48 cycles
stall injected at cycle 100 -> watchdog must fire by cycle 100 + 48 = 148
timestamp budget: sync jitter +/-1 + CDC uncertainty 1 + counter skew 1 = +/-3 cyclesCorrelate per-hop queue and credit behavior with destination service-time counters and ingress acceptance rate. Queues that grow while the sink idles name the fabric; queues that grow while the sink stalls name the endpoint.
A strong runbook entry records trigger condition, capture window, decode path, confidence criteria, and rollback-safe mitigation. Anything shorter is a request for more logs.
Per-hop queue plus credit-return behavior, against destination service time plus ingress acceptance. Sink idle with credit trouble means fabric; sink busy means endpoint.
The spike is over before capture starts. Slope precursors open the window early, so source, hotspot, and destination traces all cover the onset.