Memory Bandwidth and Throughput Limits — Inputs & Outputs

Inputs required

• Workload or trace from product/performance team

• Architecture model or RTL performance setup

• PPA budgets and software-visible constraints

Outputs produced

• Architecture decision memo

• Metric dashboard for review

• Annotated risks for RTL, verification, software, and PD

Handoff owners

• architecture owner

• performance lead

• RTL / verification / software owner as needed

Production handoff contract

Treat Memory Bandwidth and Throughput Limits inputs as a signed contract between architecture, RTL, verification, software, performance, PD, and product owners. A 10+ year engineer blocks decisions when the contract is ambiguous instead of burning weeks on invalid comparisons.

HANDOFF MANIFEST
  workload_suite: <benchmarks, traces, production scenarios>
  model_tag: <spreadsheet / simulator / RTL / emulation / silicon tag>
  metric_contract: <IPC, MPKI, bandwidth, latency, power, area>
  architecture_assumptions: <cache sizes, line size, NoC topology, coherency mode>
  owner_of_truth: <architecture / performance / RTL / software owner>
  known_risks: <unmodeled effects, missing workloads, verification concerns>

Senior acceptance rules

1. Reject mismatched workload, model, PMU, or RTL tags before comparing metrics.

2. Record the owner for every assumption that is not locally provable.

3. Preserve enough metadata that another engineer can reproduce the experiment in six months.

Architecture input diagram

INPUT CONTRACT

workload suite ─┐
PMU / trace  ───┼──► architecture analysis ──► decision memo
RTL/model tag ──┤
PPA budgets  ───┤
SW contract  ───┘

Missing any one input changes the meaning of the metric.

Architecture deep dive

Cache hierarchy trades area and power for AMAT and bandwidth.

Concept diagram

MEMORY HIERARCHY

Core
 ├─ L1I / L1D  (cycles: 1-4, tiny, latency critical)
 ├─ L2         (cycles: 8-20, private or cluster)
 ├─ LLC / SLC  (shared, bandwidth + coherency point)
 ├─ NoC        (queueing + arbitration)
 └─ DRAM/HBM   (large penalty, high energy)

AMAT = hit_time + miss_rate × miss_penalty
But senior analysis also asks: MLP, bandwidth, QoS, and tail latency.

Metric graph

MISS PENALTY WATERFALL

L1 hit          ██  3 cyc
L2 hit          ████████  12 cyc
LLC hit         ███████████████  32 cyc
DRAM miss       ████████████████████████████████████  180 cyc

Small MPKI can still dominate if miss penalty is huge.

Metrics and artifacts

• MPKI per level

• L2/L3 bandwidth utilization

• replacement policy stats

• prefetch accuracy

Mini case study

Doubling L2 size reduces capacity misses but IPC improves only 3% because conflict misses dominate a shared workload. Fix data layout and false sharing before more SRAM.

Debug branches

• If MPKI high but bandwidth low, footprint may exceed capacity.

• If bandwidth saturated, coherency or DMA may be the real limit.

Senior review question

Ask: what single metric would prove this concept is working or failing on your workload?

Key takeaways

• Connect every architecture claim to a workload and measurable metric.

• State verification and PPA impact before proposing design changes.

Common pitfalls

• Feature-driven design without MPKI/IPC/bandwidth evidence.

• Ignoring coherency and NoC traffic in cache and accelerator sizing.

Study notes

Re-read this topic with one concrete workload.