AI Accelerator Design · All levels
On-Chip Memory Hierarchy: Tricky Q&A
Senior interview and review questions for On-Chip Memory Hierarchy.
Section Q&A bank
Use these drills after completing all topics in On-Chip Memory Hierarchy. Answer with workload context, mechanism proof, artifact, owner, and release decision.
Why does adding more on-chip SRAM not always improve accelerator throughput?
diagram
[INT][ACCEL][ON-CHIP-MEMORY-HIERARCHY]
Q: Why does adding more on-chip SRAM not always improve accelerator throughput?
A:
Extra SRAM helps only when data placement, tiling, and refill policy let workloads actually reuse that capacity. If buffers are poorly partitioned or refill cadence is mismatched, larger arrays can still stall on refetch and eviction churn. Throughput tracks effective reuse and feed efficiency, not raw SRAM size.
FOLLOW-UP TRAP: Assuming SRAM capacity growth alone guarantees higher performance.When should a kernel prefer scratchpad control over cache-based access?
diagram
[INT][ACCEL][ON-CHIP-MEMORY-HIERARCHY]
Q: When should a kernel prefer scratchpad control over cache-based access?
A:
Scratchpads are usually better when access patterns are regular, synchronization points are known, and software can schedule movement explicitly. Cache paths are often better for irregular or rapidly changing locality where static planning is brittle. The decision should come from measured miss or spill behavior and engineering complexity tradeoffs.
FOLLOW-UP TRAP: Choosing scratchpad or cache by default without locality measurements.How do teams avoid overestimating available HBM bandwidth during design planning?
diagram
[INT][ACCEL][ON-CHIP-MEMORY-HIERARCHY]
Q: How do teams avoid overestimating available HBM bandwidth during design planning?
A:
They model real traffic shape, concurrency, arbitration overhead, and controller behavior instead of relying on peak spec bandwidth. Planning includes burst efficiency, read/write mix, and contention from simultaneous kernels to estimate sustainable delivery. Guardbands and stress scenarios are essential before committing compute sizing decisions.
FOLLOW-UP TRAP: Sizing compute arrays from theoretical HBM peak numbers alone.What is a common symptom of memory bank conflicts in on-chip SRAM systems?
diagram
[INT][ACCEL][ON-CHIP-MEMORY-HIERARCHY]
Q: What is a common symptom of memory bank conflicts in on-chip SRAM systems?
A:
You often see high compute idle time even though total memory bandwidth looks adequate, because requests serialize on a few hot banks. Profilers usually show elevated bank-conflict counters and periodic stall bursts tied to specific access strides. Layout remapping or stride padding typically restores parallelism.
FOLLOW-UP TRAP: Diagnosing conflict stalls as generic bandwidth shortage without bank-level traces.Q&A drill guide
diagram
SYMPTOM -> BOTTLENECK CLASS -> ARTIFACT -> OWNER ACTION -> VALIDATIONSketch while answering
diagram
MEMORY HIERARCHY VIEW
register/SRAM -> shared buffers -> NoC -> HBM
locality quality decides how long compute stays fedCommon traps
Separate locality vs bandwidth limits
Quantify bank conflicts
Tune tiling before resizing compute arrays
Key takeaways
Tie every accelerator claim to a reproducible workload slice and one primary metric trend.
Prefer bounded fixes with clear owner and rollback boundary over broad tuning bundles.
Common pitfalls
Optimizing synthetic kernels without production-shape validation.
Reading average latency while ignoring p95 and p99 behavior.
Declaring sparse or precision wins without fallback and quality evidence.