Computer Architecture · All levels
Memory Hierarchy Tricky Q&A
20+ senior Memory Hierarchy interview questions.
Q&A bank
Answer with mechanism, pitfall, validation check, and product judgment. At 10+ years, a correct definition is not enough.
Senior answer rubric
Start with the failing metric and analysis context.
Explain the microarchitectural or system mechanism.
Name the cheapest evidence-gathering experiment.
Choose a bounded fix and state what it can regress.
Close with product tradeoff, validation, or escalation criteria.
Why is miss rate alone insufficient for memory hierarchy decisions?
diagram
[INT][ARCH]
Q: Why is miss rate alone insufficient for memory hierarchy decisions?
A:
Performance impact depends on miss penalty, MLP, and hit-latency shifts; lower miss rate can still hurt IPC if latency rises.
FOLLOW-UP TRAP: Optimizing only miss rate.How do you prioritize cache organization candidates?
diagram
[INT][ARCH]
Q: How do you prioritize cache organization candidates?
A:
Score options by product objective function: IPC/latency/perf-watt with area and implementation feasibility constraints.
FOLLOW-UP TRAP: Ranking by single synthetic benchmark.What indicates associativity increase is not worth it?
diagram
[INT][ARCH]
Q: What indicates associativity increase is not worth it?
A:
Marginal MPKI gain with measurable hit-latency, power, or tail-latency regression on target workloads.
FOLLOW-UP TRAP: Keeping higher associativity by default.How do you detect replacement-policy thrashing quickly?
diagram
[INT][ARCH]
Q: How do you detect replacement-policy thrashing quickly?
A:
Use set-level pressure heatmaps and short line-lifetime spikes coupled with MPKI bursts.
FOLLOW-UP TRAP: Inspecting only aggregate cache misses.Why track prefetch pollution separately from demand misses?
diagram
[INT][ARCH]
Q: Why track prefetch pollution separately from demand misses?
A:
Policy fixes differ; pollution requires admission/timeliness tuning, while demand misses may need structure or mapping changes.
FOLLOW-UP TRAP: Combining all misses into one bucket.What is a fair rollout strategy for adaptive replacement?
diagram
[INT][ARCH]
Q: What is a fair rollout strategy for adaptive replacement?
A:
Stage deployment with telemetry and fallback, validating fairness and p95 latency per tenant class.
FOLLOW-UP TRAP: Global rollout without guardrails.How do you locate hidden bandwidth loss points?
diagram
[INT][ARCH]
Q: How do you locate hidden bandwidth loss points?
A:
Build per-hop bandwidth waterfall and queue-latency correlation from core to DRAM.
FOLLOW-UP TRAP: Jumping directly to DRAM timing tuning.When does low DRAM utilization still indicate a memory problem?
diagram
[INT][ARCH]
Q: When does low DRAM utilization still indicate a memory problem?
A:
When upstream bottlenecks or scheduling inefficiencies prevent requests from reaching DRAM effectively.
FOLLOW-UP TRAP: Assuming unused DRAM implies headroom everywhere.How do you validate NoC QoS tuning for memory traffic?
diagram
[INT][ARCH]
Q: How do you validate NoC QoS tuning for memory traffic?
A:
Measure throughput plus p95/p99 latency and fairness across mixed read/write and tenant workloads.
FOLLOW-UP TRAP: Using average bandwidth only.What is the biggest risk in controller scheduling tweaks?
diagram
[INT][ARCH]
Q: What is the biggest risk in controller scheduling tweaks?
A:
Improving aggregate GB/s while creating starvation or latency cliffs for critical flows.
FOLLOW-UP TRAP: Ignoring QoS regressions.How do you differentiate coherence storm from ordinary contention?
diagram
[INT][ARCH]
Q: How do you differentiate coherence storm from ordinary contention?
A:
Look for abnormal replay/retry/invalidate event surges and protocol-state anomalies beyond normal queue pressure patterns.
FOLLOW-UP TRAP: Calling every event burst contention.Why should cache debug include power-state transitions?
diagram
[INT][ARCH]
Q: Why should cache debug include power-state transitions?
A:
Restore/order boundaries can expose stale-state races that never appear in always-on testing.
FOLLOW-UP TRAP: Skipping low-power scenarios.What is a minimum evidence set for cache debug review?
diagram
[INT][ARCH]
Q: What is a minimum evidence set for cache debug review?
A:
Deterministic repro script, PMU deltas, protocol trace window, invariant check result, and validated fix regression.
FOLLOW-UP TRAP: Presenting only anecdotal symptoms.How do you bound risk when changing coherency logic late?
diagram
[INT][ARCH]
Q: How do you bound risk when changing coherency logic late?
A:
Prefer minimal localized invariant-preserving patch, with focused regression on affected states and traffic classes.
FOLLOW-UP TRAP: Large refactor near release.Why is reuse-distance analysis valuable for replacement tuning?
diagram
[INT][ARCH]
Q: Why is reuse-distance analysis valuable for replacement tuning?
A:
It reveals whether evicted lines had near-term reuse, guiding insertion/victim policy beyond surface MPKI metrics.
FOLLOW-UP TRAP: Relying only on LRU intuition.When should architecture escalate memory issue to SoC team?
diagram
[INT][ARCH]
Q: When should architecture escalate memory issue to SoC team?
A:
When root cause involves shared NoC arbitration, firmware QoS policy, or cross-IP traffic contracts outside cache local control.
FOLLOW-UP TRAP: Trying to force local cache-only fix.How do you avoid overfitting memory policy to one workload?
diagram
[INT][ARCH]
Q: How do you avoid overfitting memory policy to one workload?
A:
Require gains across representative portfolio and verify no unacceptable regressions in antagonistic patterns.
FOLLOW-UP TRAP: Tuning for a flagship benchmark only.What makes a memory hierarchy metric dashboard executive-useful?
diagram
[INT][ARCH]
Q: What makes a memory hierarchy metric dashboard executive-useful?
A:
It ties counters to product outcomes: throughput, tail latency, perf/watt, and release risk with ownership.
FOLLOW-UP TRAP: Dumping raw counters without decision framing.How do you explain cache fairness to non-architecture stakeholders?
diagram
[INT][ARCH]
Q: How do you explain cache fairness to non-architecture stakeholders?
A:
Fairness means one workload cannot monopolize shared cache and violate latency/QoS commitments for others.
FOLLOW-UP TRAP: Using only microarchitectural jargon.How do senior engineers close memory hierarchy investigations?
diagram
[INT][ARCH]
Q: How do senior engineers close memory hierarchy investigations?
A:
They ship a mechanism-backed fix, telemetry guardrails, regression coverage, and explicit signoff criteria tied to product KPIs.
FOLLOW-UP TRAP: Closing with 'metrics look better now'.Q&A drill guide
diagram
MECHANISM → METRIC → EXPERIMENT → TRADEOFF → VALIDATIONSketch while answering
diagram
MEMORY HIERARCHY
Core
├─ L1I / L1D (cycles: 1-4, tiny, latency critical)
├─ L2 (cycles: 8-20, private or cluster)
├─ LLC / SLC (shared, bandwidth + coherency point)
├─ NoC (queueing + arbitration)
└─ DRAM/HBM (large penalty, high energy)
AMAT = hit_time + miss_rate × miss_penalty
But senior analysis also asks: MLP, bandwidth, QoS, and tail latency.Key takeaways
Connect every architecture claim to a workload and measurable metric.
State verification and PPA impact before proposing design changes.
Common pitfalls
Feature-driven design without MPKI/IPC/bandwidth evidence.
Ignoring coherency and NoC traffic in cache and accelerator sizing.