Cache Organization and Access Path — Extended Case Study

Extended case study

A review is called because a workload regresses after a Cache Organization and Access Path change.

Background

A stable baseline existed until a Memory Hierarchy change improved one benchmark and regressed a product workload on Cache hierarchy KPI dashboard.

Symptoms observed

• Regression in Cache hierarchy KPI dashboard

• Sim vs silicon disagreement

• Pressure to revert or ship risk

Investigation timeline

1. Freeze tags

2. Reproduce

3. Cluster

4. Experiment

5. Validate

6. Memo

Root cause

A hidden assumption in Cache Organization and Access Path failed under an unrepresented workload phase.

Fix and validation

• Confirm hit/miss counter definitions and sampling windows are identical across variants.

• Decompose miss changes into conflict vs capacity to understand true associativity benefit.

• Measure hit-latency shift impact on core stall cycles and tail latency percentiles.

• Run power/perf sweep across representative SKU workloads.

• Decide with objective function (throughput, latency, perf/watt) per product segment.

Lessons learned

• Workload coverage beats clever microarchitecture

• Every change needs rollback triggers

CACHE ORG TRADEOFF
workload: mixed_cloud_frontend
l1d_hit_rate_pct: 95.8
l2_hit_rate_pct_before: 87.1
l2_hit_rate_pct_after: 90.4
l2_hit_latency_cycles: 13 -> 16
llc_mpki: 11.9 -> 10.7
e_per_instruction_nj: 1.82 -> 1.98
ipc: 1.46 -> 1.44
action: keep 12-way pilot only for latency-insensitive SKU; evaluate way-prediction

Architecture deep dive

Cache hierarchy trades area and power for AMAT and bandwidth.

Concept diagram

MEMORY HIERARCHY

Core
 ├─ L1I / L1D  (cycles: 1-4, tiny, latency critical)
 ├─ L2         (cycles: 8-20, private or cluster)
 ├─ LLC / SLC  (shared, bandwidth + coherency point)
 ├─ NoC        (queueing + arbitration)
 └─ DRAM/HBM   (large penalty, high energy)

AMAT = hit_time + miss_rate × miss_penalty
But senior analysis also asks: MLP, bandwidth, QoS, and tail latency.

Metric graph

MISS PENALTY WATERFALL

L1 hit          ██  3 cyc
L2 hit          ████████  12 cyc
LLC hit         ███████████████  32 cyc
DRAM miss       ████████████████████████████████████  180 cyc

Small MPKI can still dominate if miss penalty is huge.

Metrics and artifacts

• MPKI per level

• L2/L3 bandwidth utilization

• replacement policy stats

• prefetch accuracy

Mini case study

Doubling L2 size reduces capacity misses but IPC improves only 3% because conflict misses dominate a shared workload. Fix data layout and false sharing before more SRAM.

Debug branches

• If MPKI high but bandwidth low, footprint may exceed capacity.

• If bandwidth saturated, coherency or DMA may be the real limit.

Senior review question

Ask: what single metric would prove this concept is working or failing on your workload?

Key takeaways

• Connect every architecture claim to a workload and measurable metric.

• State verification and PPA impact before proposing design changes.

Common pitfalls

• Feature-driven design without MPKI/IPC/bandwidth evidence.

• Ignoring coherency and NoC traffic in cache and accelerator sizing.

Study notes

Re-read this topic with one concrete workload.