Memory Locality and Hierarchy Co-Design — Inputs & Outputs

Inputs required

• Hierarchy capacities and latencies

• Workload footprint traces

• Coherency and consistency requirements

Outputs produced

• Locality plan

• Hierarchy pressure forecasts

• Energy-performance tradeoff table

Handoff owners

• architecture owner

• performance lead

• RTL / verification / software owner as needed

Production handoff contract

Treat Memory Locality and Hierarchy Co-Design inputs as a signed contract between architecture, RTL, verification, software, performance, PD, and product owners. A 10+ year engineer blocks decisions when the contract is ambiguous instead of burning weeks on invalid comparisons.

HANDOFF MANIFEST
  workload_suite: <benchmarks, traces, production scenarios>
  model_tag: <spreadsheet / simulator / RTL / emulation / silicon tag>
  metric_contract: <IPC, MPKI, bandwidth, latency, power, area>
  architecture_assumptions: <cache sizes, line size, NoC topology, coherency mode>
  owner_of_truth: <architecture / performance / RTL / software owner>
  known_risks: <unmodeled effects, missing workloads, verification concerns>

Senior acceptance rules

1. Reject mismatched workload, model, PMU, or RTL tags before comparing metrics.

2. Record the owner for every assumption that is not locally provable.

3. Preserve enough metadata that another engineer can reproduce the experiment in six months.

Architecture input diagram

INPUT CONTRACT

workload suite ─┐
PMU / trace  ───┼──► architecture analysis ──► decision memo
RTL/model tag ──┤
PPA budgets  ───┤
SW contract  ───┘

Missing any one input changes the meaning of the metric.

Architecture deep dive

Accelerators win on locality and bandwidth contracts, not peak OPS alone.

Concept diagram

ACCELERATOR DATAFLOW

Host CPU ── commands ──► Queue / scheduler
   ▲                         │
   │ completion              ▼
Coherent memory ◄── DMA ── Local SRAM ──► Compute array
                         ▲       │
                         └ tiles ┘

Peak TOPS matters only when data reaches the array at the needed rate.

Metric graph

UTILIZATION BREAKDOWN

compute active   ██████████████████  58%
DMA wait         ██████████          31%
host sync        █████               15%
cache/coherency  ████                12%
idle bubbles     ███████             22%

Low utilization is usually a system integration problem.

Metrics and artifacts

• accelerator utilization

• DMA bandwidth

• kernel launch overhead

• coherency invalidation rate

Mini case study

NPU met TOPs target but end-to-end inference slow — DMA and weight fetch dominated. Architecture added on-chip SRAM tile and double-buffering.

Debug branches

• If util low, check launch overhead and host sync first.

• If BW high, examine weight layout and sparsity support.

Senior review question

Ask: what single metric would prove this concept is working or failing on your workload?

Key takeaways

• Connect every architecture claim to a workload and measurable metric.

• State verification and PPA impact before proposing design changes.

Common pitfalls

• Feature-driven design without MPKI/IPC/bandwidth evidence.

• Ignoring coherency and NoC traffic in cache and accelerator sizing.

Study notes

Re-read this topic with one concrete workload.