DenseChip

Computer Architecture · All levels

Replacement Policy and Miss Behavior

Replacement Policy and Miss Behavior — computer architecture for silicon teams.

On-call / interview prompt

Production traces show sudden LLC MPKI spikes after firmware update, mostly in mixed tenant workloads. How do you prove policy pathologies versus workload shift?

diagram
ARCHITECTURE ANALYSIS CHAIN

1. METRIC     — IPC, CPI, MPKI, bandwidth, latency, queue depth, stall cycles
2. HYPOTHESIS — microarch or system cause ordered by likelihood
3. EXPERIMENT — trace, PMU counter, simulation, or RTL probe
4. CHANGE      — pipeline, cache, NoC, or memory hierarchy adjustment
5. VALIDATION  — workload replay, regression suite, PPA impact

Topic overview

Use replacement and insertion policies to minimize destructive eviction patterns while preserving fairness and bandwidth efficiency across tenants and cores.

Mechanism to narrate

  • Section: Memory Hierarchy

  • Primary artifact: LLC miss decomposition + reuse distance report

  • Downstream dependency: SoC QoS enforcement, firmware scheduling, and customer-perceived performance stability rely on miss behavior.

Staff/principal ownership model

Own Replacement Policy and Miss Behavior as a product architecture decision, not a page of notes. A senior architect names the metric, the mechanism, the cross-team dependency, and the smallest evidence-producing experiment.

diagram
STAFF ARCHITECTURE REVIEW MEMO — Memory Hierarchy / Replacement Policy and Miss Behavior

1. Current state
   - Failing / watched metric: LLC miss decomposition + reuse distance report
   - Workload / benchmark / trace: <fill before review>
   - Model tag, RTL tag, simulator version, PMU setup: <fill before review>
   - Scope: core, cache level, NoC path, coherency domain, accelerator, or SoC budget

2. Root-cause hypothesis
   - Most likely mechanism: <name pipeline/cache/NoC/coherency/perf mechanism>
   - Competing hypothesis: <name the second plausible cause>
   - Evidence still missing: <counter, trace, waveform, model sweep, or workload slice>

3. Proposed action
   - Minimal reversible change: <microarchitecture, policy, sizing, traffic, or software contract change>
   - Expected improvement: <metric delta>
   - Regression risk: Mis-tuned replacement can cause latency SLO violations and noisy-neighbor behavior in shared-cache systems.

4. Regression and signoff
   - Re-run: LLC miss decomposition + reuse distance report
   - Must not regress: SoC QoS enforcement, firmware scheduling, and customer-perceived performance stability rely on miss behavior.
   - Decision owner: architecture owner

Sub-lessons in this topic

  1. mechanism — Mechanism

  2. inputs-outputs — Inputs & Outputs

  3. reports — Reports & Metrics

  4. debug-playbook — Debug Playbook

  5. worked-example — Worked Example

  6. pitfalls — Pitfalls & Red Flags

  7. interview — Interview Drills

  8. checklist — Review Checklist

  9. theory-deep-dive — Theory Deep Dive

  10. design-space — Design Space Exploration

  11. case-study-expanded — Extended Case Study

  12. step-by-step-walkthrough — Step-by-Step Walkthrough

  13. comparison-matrix — Comparison Matrix

  14. software-programmer-view — Software / Programmer View

  15. silicon-ppa-impact — Silicon & PPA Impact

Related topics

Key takeaways

  • Master Replacement Policy and Miss Behavior through workload metrics, not feature lists.

Architecture deep dive

Cache hierarchy trades area and power for AMAT and bandwidth.

Concept diagram

diagram
MEMORY HIERARCHY

Core
 ├─ L1I / L1D  (cycles: 1-4, tiny, latency critical)
 ├─ L2         (cycles: 8-20, private or cluster)
 ├─ LLC / SLC  (shared, bandwidth + coherency point)
 ├─ NoC        (queueing + arbitration)
 └─ DRAM/HBM   (large penalty, high energy)

AMAT = hit_time + miss_rate × miss_penalty
But senior analysis also asks: MLP, bandwidth, QoS, and tail latency.

Metric graph

diagram
MISS PENALTY WATERFALL

L1 hit          ██  3 cyc
L2 hit          ████████  12 cyc
LLC hit         ███████████████  32 cyc
DRAM miss       ████████████████████████████████████  180 cyc

Small MPKI can still dominate if miss penalty is huge.

Metrics and artifacts

  • MPKI per level

  • L2/L3 bandwidth utilization

  • replacement policy stats

  • prefetch accuracy

Mini case study

Doubling L2 size reduces capacity misses but IPC improves only 3% because conflict misses dominate a shared workload. Fix data layout and false sharing before more SRAM.

Debug branches

  • If MPKI high but bandwidth low, footprint may exceed capacity.

  • If bandwidth saturated, coherency or DMA may be the real limit.

Senior review question

Ask: what single metric would prove this concept is working or failing on your workload?

Key takeaways

  • Connect every architecture claim to a workload and measurable metric.

  • State verification and PPA impact before proposing design changes.

Common pitfalls

  • Feature-driven design without MPKI/IPC/bandwidth evidence.

  • Ignoring coherency and NoC traffic in cache and accelerator sizing.

Practice this lesson