Memory Bandwidth and Throughput Limits — Interview Drills

[INT][ARCH][TOPIC]

Q: Why can memory stalls grow even when DRAM utilization looks moderate?

A:
Effective bottleneck may exist earlier in hierarchy: NoC arbitration, controller queueing, or bank conflicts can throttle delivered bandwidth.

FOLLOW-UP TRAP: Using DRAM utilization as sole bottleneck metric.

[INT][ARCH][TOPIC]

Q: How do you quantify sustained vs peak bandwidth value?

A:
Use delivered GB/s over representative workload windows plus latency percentiles, not synthetic peak bursts alone.

FOLLOW-UP TRAP: Reporting only benchmark max throughput.

[INT][ARCH][TOPIC]

Q: What is a safe way to tune write-drain policy?

A:
Run bounded sweeps with read-latency and fairness guards; accept only settings that improve throughput without violating tail-latency targets.

FOLLOW-UP TRAP: Maximizing throughput while ignoring latency SLO.

[INT][ARCH][STAFF]

Q: You inherit a late-stage Memory Bandwidth and Throughput Limits failure one week before release. What do you do in the first hour?

A:
Freeze the workload/model/RTL tag, name the failing metric (Bandwidth waterfall + queue latency report), confirm counter setup, cluster the issue by structure or workload phase, assign the first experiment, and publish a validation/owner plan before changing architecture.

FOLLOW-UP TRAP: Jumping directly to a larger cache, wider pipe, or extra NoC link without preserving evidence.

[INT][ARCH][STAFF]

Q: When would you stop trying to improve Memory Bandwidth and Throughput Limits and escalate?

A:
Escalate when the remaining risk crosses ownership boundaries, consumes shared margin, changes signed-off assumptions, or threatens CPU/GPU scheduling policy, SoC NoC tuning, and customer workload scaling depend on reliable bandwidth delivery.. Bring exact report lines and options, not vague concern.

FOLLOW-UP TRAP: Escalating without data or continuing alone after a cross-team decision is needed.

VISUAL MODEL — Memory Hierarchy / Memory Bandwidth and Throughput Limits

        workload / trace
              │
              ▼
   metric symptom (Bandwidth waterfall + queue latency report)
              │
              ▼
     likely microarchitectural mechanism
              │
      ┌───────┼────────┐
      ▼       ▼        ▼
  pipeline  memory    fabric/coherency
  stalls    misses    queues / ordering
      │       │        │
      └───────┼────────┘
              ▼
        bounded design change
              │
              ▼
   validation workload + PPA regression

MEMORY HIERARCHY

Core
 ├─ L1I / L1D  (cycles: 1-4, tiny, latency critical)
 ├─ L2         (cycles: 8-20, private or cluster)
 ├─ LLC / SLC  (shared, bandwidth + coherency point)
 ├─ NoC        (queueing + arbitration)
 └─ DRAM/HBM   (large penalty, high energy)

AMAT = hit_time + miss_rate × miss_penalty
But senior analysis also asks: MLP, bandwidth, QoS, and tail latency.

MISS PENALTY WATERFALL

L1 hit          ██  3 cyc
L2 hit          ████████  12 cyc
LLC hit         ███████████████  32 cyc
DRAM miss       ████████████████████████████████████  180 cyc

Small MPKI can still dominate if miss penalty is huge.