AI Accelerator Design · All levels

AI Accelerator Metrics Reference

Core metrics for throughput, latency, quality, utilization, and sustained silicon efficiency.

Core metric families

Throughput and latency: tokens/s or inferences/s plus p95/p99 tails.
Utilization: compute occupancy, tensor-core or array fill efficiency.
Memory: bandwidth efficiency, reuse ratio, and stall fractions.
Quality: accuracy or loss movement under precision and scheduling changes.
Silicon: perf-per-watt, thermal throttle duty cycle, and frequency residency.

diagram

BANDWIDTH LENS - AI Accelerator closure

working-set pressure
  ^
  |                saturation zone
  |          ----------------------------
  |      o   unstable tail latency
  |   o      tuning candidate
  | o        baseline behavior
  +-------------------------------------> optimization iteration

Primary metric tracked:
throughput / p99 / perf-per-watt

Practice this lesson