Extended case study

Integration review: request retry rate, directory occupancy, p99 fabric latency regresses after a change touching CHI Topology Basics.

Background

Baseline traffic passed compliance and performance targets. A bridge update, firmware change, or clock/reset tweak introduced intermittent failures visible only under mixed traffic.

Symptoms observed

• Regression in request retry rate, directory occupancy, p99 fabric latency

• VIP warning followed by software timeout (symptom lag)

• Directed tests pass; stress or product replay fails

• Two teams disagree because they look at different layers

Investigation timeline

1. Hour 0: freeze sim tag, firmware, and spec revision

2. Hour 1: capture first failing transaction with ID/address

3. Hour 2: correlate waveform, VIP monitor, and counter

4. Hour 3: classify: rule violation vs config vs timing vs load

5. Hour 4: reduce to 3-transaction minimal sequence

6. Hour 5: bounded RTL or register fix + regression list

7. Hour 6: compliance replay + product workload signoff memo

Root cause

The failing behavior traced to a violated assumption in CHI Topology Basics: CHI separates request, response, data, and snoop flows across nodes with directory and home-node responsibilities.

Fix and validation

• Minimal reversible change at the owning boundary

• Re-run CHI node map, VC utilization report, request/response correlation log on failing and baseline seeds

• Compliance suite + mixed-traffic regression

• Document software-visible impact and waiver if any

Lessons learned

• First bad transaction beats loudest timeout

• Layer alignment across RTL, VIP, firmware, and analyzer

• Performance and correctness regressions need separate evidence

CASE STUDY METRICS — CHI Topology Basics

baseline     request retry rate, directory occupancy, p99 fabric latency: within target
regressed    request retry rate, directory occupancy, p99 fabric latency: fails product threshold
after fix    request retry rate, directory occupancy, p99 fabric latency: restored + compliance PASS
residual risk: document waiver or monitor in field

Sequence under stress

SEQUENCE — CHI Topology Basics

  initiator            interconnect/PHY            target
      |  request (id) ------->  |                     |
      |                         |  forward ----------> |
      |                         |                     | work
      |                         |  <---- response ---- |
      |  <----- complete ------ |                     |
      |
   metric captured here: request retry rate, directory occupancy, p99 fabric latency

Protocol deep dive

Coherence extends memory transactions with snoop and state — traffic multiplies when software shares cache lines.

Concept diagram

COHERENCE TRAFFIC FLOW

RN issues coherent read
   -> HN looks up directory
   -> snoops to sharers
   -> data + state update returned

False sharing: different variables, same cache line -> coherence storm.

Metric graph

COHERENCY TRAFFIC STACK

data fetch        ████████
snoop responses   ██████████████
writebacks        ██████
maintenance ops   ████

High snoop stack with good IPC -> suspect line sharing before faster NoC.

Metrics and artifacts to collect

• snoop rate

• intervention latency

• coherency transaction mix

• false sharing indicators

Mini case study

Benchmark IPC looked fine but system power spiked: per-core counters were on one cache line. Padding counters fixed coherency traffic without any NoC change.

Debug branches

• If snoop latency high, check home node placement and directory policy.

• If ordering bug, run litmus sequences before microarch changes.

• If traffic storm, profile cache line sharing in software layout.

Senior review question

Ask: what is the first transaction that deviates, and which spec rule does it test?

Key takeaways

• Connect every protocol claim to a transaction identity and measurable metric.

• Store the artifact (waveform, log, counter) next to every signoff decision.

Common pitfalls

• Debugging timeouts without finding the first bad transaction.

• Quoting peak bus width without payload efficiency and retry overhead.

• Treating VIP compliance as a substitute for system integration replay.

Field case notes

Mixed traffic exposed a bug that single-master directed tests missed for three weeks.