Computer Architecture · All levels

Cache Organization and Access Path — Worked Example

Worked Example for Cache Organization and Access Path (Memory Hierarchy).

Scenario

A product workload exposes a Cache Organization and Access Path issue late in architecture review.

Timeline

Metric fails at review meeting
Engineer captures metric report, trace snippet, and workload phase
Root cause traced to incorrect assumption from prior stage
Minimal architecture change or policy experiment applied and documented
Workload regression matrix re-run on the tagged model

Outcome

Architecture decision accepted with documented metric movement, risk, and validation proof.

Senior debrief

After solving the example, write the debrief a lead would expect: what changed, why it worked, what could regress, and what permanent methodology update prevents recurrence.

diagram

STAFF ARCHITECTURE REVIEW MEMO — Memory Hierarchy / Cache Organization and Access Path

1. Current state
   - Failing / watched metric: Cache hierarchy KPI dashboard
   - Workload / benchmark / trace: <fill before review>
   - Model tag, RTL tag, simulator version, PMU setup: <fill before review>
   - Scope: core, cache level, NoC path, coherency domain, accelerator, or SoC budget

2. Root-cause hypothesis
   - Most likely mechanism: <name pipeline/cache/NoC/coherency/perf mechanism>
   - Competing hypothesis: <name the second plausible cause>
   - Evidence still missing: <counter, trace, waveform, model sweep, or workload slice>

3. Proposed action
   - Minimal reversible change: <microarchitecture, policy, sizing, traffic, or software contract change>
   - Expected improvement: <metric delta>
   - Regression risk: Overbuilt cache structures can miss perf/watt and area budgets while adding coherence and validation burden.

4. Regression and signoff
   - Re-run: Cache hierarchy KPI dashboard
   - Must not regress: Pipeline stall behavior, NoC traffic profile, and SoC thermal budget depend on cache organization.
   - Decision owner: architecture owner

Before / after metric graph

diagram

METRIC TREND GRAPH — Cache Organization and Access Path

IPC / throughput
  ^
  |                  target
  |                 ─ ─ ─ ─ ─ ─ ─
  |            ● after bounded fix
  |         /
  |    ● baseline
  |  /
  |● failing run
  +---------------------------------> experiment index
    bad tag       hypothesis        accepted fix

Readout rule:
  - one dot is not a conclusion
  - compare against same workload, seed, model tag, and counter setup
  - explain why the fix moved the metric, not just that it moved

Architecture deep dive

Cache hierarchy trades area and power for AMAT and bandwidth.

Concept diagram

diagram

MEMORY HIERARCHY

Core
 ├─ L1I / L1D  (cycles: 1-4, tiny, latency critical)
 ├─ L2         (cycles: 8-20, private or cluster)
 ├─ LLC / SLC  (shared, bandwidth + coherency point)
 ├─ NoC        (queueing + arbitration)
 └─ DRAM/HBM   (large penalty, high energy)

AMAT = hit_time + miss_rate × miss_penalty
But senior analysis also asks: MLP, bandwidth, QoS, and tail latency.

Metric graph

diagram

MISS PENALTY WATERFALL

L1 hit          ██  3 cyc
L2 hit          ████████  12 cyc
LLC hit         ███████████████  32 cyc
DRAM miss       ████████████████████████████████████  180 cyc

Small MPKI can still dominate if miss penalty is huge.

Metrics and artifacts

MPKI per level
L2/L3 bandwidth utilization
replacement policy stats
prefetch accuracy

Mini case study

Doubling L2 size reduces capacity misses but IPC improves only 3% because conflict misses dominate a shared workload. Fix data layout and false sharing before more SRAM.

Debug branches

If MPKI high but bandwidth low, footprint may exceed capacity.
If bandwidth saturated, coherency or DMA may be the real limit.

Senior review question

Ask: what single metric would prove this concept is working or failing on your workload?

Key takeaways

Connect every architecture claim to a workload and measurable metric.
State verification and PPA impact before proposing design changes.

Common pitfalls

Feature-driven design without MPKI/IPC/bandwidth evidence.
Ignoring coherency and NoC traffic in cache and accelerator sizing.

Study notes

Re-read this topic with one concrete workload.

Practice this lesson