The NPU reaches peak TOPs in isolation, yet end-to-end inference misses target by a wide margin. Isolation hides transfers, launch and sync.
Assume 16 GOP of work, 32 TOPS of peak compute, 8 MB of weights, 16 GB/s of DMA and 0.15 ms of launch plus sync. Target latency is 0.8 ms.
0-5 min: define the end-to-end metric and the exact workload.
5-15 min: split wall time into compute, bytes and overhead.
15-25 min: rank tiling, double buffering and contract hypotheses.
25-35 min: propose SRAM tile, DMA and software contract options.
35-45 min: state which metric is safe to publish and which misleads.
Compute time is 16 GOP / 32 TOPS = 0.5 ms. Weight time is 8 MB / 16 GB/s = 0.5 ms.
Naive total is 0.5 + 0.5 + 0.15 = 1.15 ms against a 0.8 ms target, about 44 percent over. The accelerator is active 0.5 / 1.15 = 0.43, about 43 percent.
Compute: 16 GOP / 32 TOPS = 0.5 ms
Bytes: 8 MB / 16 GB/s = 0.5 ms
Naive: 0.5 + 0.5 + 0.15 = 1.15 ms vs 0.8 ms target
Over: 1.15 / 0.8 = 1.44, about 44 percent over target
Active: 0.5 / 1.15 = 0.43, about 43 percent utilizationActive fraction: compute time over end-to-end time. A low fraction moves the debug to bytes and overhead, not the array.
Four tiles cut each transfer to 0.125 ms. Double buffering overlaps each tile fetch with the previous tile compute.
Pipelined total is 0.125 fill + 0.5 compute + 0.15 overhead = 0.775 ms, inside the 0.8 ms target. The fix is schedule and SRAM, not a faster array.
Schedule | Time (ms) | Meets 0.8 ms |
|---|---|---|
Naive serial | 1.15 | No |
Tiled, no overlap | 1.15 | No |
Tiled double-buffered | 0.775 | Yes |
Peak TOPs describe the array, not the workload. The publishable number is sustained throughput at the measured arithmetic intensity.
Name the contract: tile size, DMA reservation and launch budget. Marketing gets the roofline point, not the peak.
Bytes are hidden, so the array itself is the bound. Now array throughput or intensity is the lever.