DenseChip

Computer Architecture · All levels

Memory Hierarchy

Senior memory hierarchy registry for silicon engineers: cache organization, miss behavior, bandwidth scaling, and cache debug from simulation to silicon.

Section goal

Drive cache and memory decisions using MPKI, miss penalty, bandwidth efficiency, and latency QoS evidence across product workloads.

Mechanism to narrate

  • Hierarchy tuning is a latency-throughput tradeoff across core, fabric, and DRAM behavior.

  • Miss rate alone is insufficient; pair it with miss penalty, MLP, and bandwidth utilization.

  • Debug should correlate PMU counters, traffic traces, and coherence state transitions before proposing structural changes.

Senior course bar for this section

  • Every topic should end with an architecture decision, not only concept recall.

  • Every fix should state expected metric movement and likely regression surface.

  • Every open assumption should have an owner, tag, and review date.

  • Every recurring issue should become a methodology guardrail or checklist item.

  1. cache-organization/ — Cache Organization and Access Path

  2. replacement-and-miss/ — Replacement Policy and Miss Behavior

  3. memory-bandwidth/ — Memory Bandwidth and Throughput Limits

  4. cache-debug/ — Cache Debug and Coherency Triage

Related topics

Key takeaways

  • At 10+ years, memory decisions are accepted only with quantified product-impact and regression boundaries.

  • Cache optimization must include coherence, QoS fairness, and power side effects.

Section deep dive

Cache hierarchy trades area and power for AMAT and bandwidth.

Concept diagram

diagram
MEMORY HIERARCHY

Core
 ├─ L1I / L1D  (cycles: 1-4, tiny, latency critical)
 ├─ L2         (cycles: 8-20, private or cluster)
 ├─ LLC / SLC  (shared, bandwidth + coherency point)
 ├─ NoC        (queueing + arbitration)
 └─ DRAM/HBM   (large penalty, high energy)

AMAT = hit_time + miss_rate × miss_penalty
But senior analysis also asks: MLP, bandwidth, QoS, and tail latency.

Metric graph

diagram
MISS PENALTY WATERFALL

L1 hit          ██  3 cyc
L2 hit          ████████  12 cyc
LLC hit         ███████████████  32 cyc
DRAM miss       ████████████████████████████████████  180 cyc

Small MPKI can still dominate if miss penalty is huge.

Metrics and artifacts

  • MPKI per level

  • L2/L3 bandwidth utilization

  • replacement policy stats

  • prefetch accuracy

Mini case study

Doubling L2 size reduces capacity misses but IPC improves only 3% because conflict misses dominate a shared workload. Fix data layout and false sharing before more SRAM.

Debug branches

  • If MPKI high but bandwidth low, footprint may exceed capacity.

  • If bandwidth saturated, coherency or DMA may be the real limit.

Senior review question

Ask: what single metric would prove this concept is working or failing on your workload?

Key takeaways

  • Connect every architecture claim to a workload and measurable metric.

  • State verification and PPA impact before proposing design changes.

Common pitfalls

  • Feature-driven design without MPKI/IPC/bandwidth evidence.

  • Ignoring coherency and NoC traffic in cache and accelerator sizing.

Practice this lesson