Only the acceleratable fraction matters. If 60% of runtime can run 8 times faster, the whole runs about 2.11 times faster, not 8.
The rest is serial ballast. It caps every proposal before RTL starts.
Amdahl: S = 1 / ((1 - f) + f / s), f = 0.6, s = 8
denominator = 0.4 + 0.6 / 8 = 0.4 + 0.075 = 0.475
S = 1 / 0.475 = 2.105... ≈ 2.11xKernel mix share | Plan | End-to-end result |
|---|---|---|
Conv 50%, attention 30%, elementwise 20% (sums to 100%) | Accelerate conv 4x only | 1 / (0.5 + 0.5 / 4) = 1 / 0.625 = 1.60x |
Same mix | Accelerate conv 4x and attention 2x | Serial 0.2 + 0.5/4 + 0.3/2 = 0.475, so 1 / 0.475 = 2.11x |
An offload wins when launch plus movement plus remote compute beats local compute. Fixed costs dominate small kernels.
Assume a 500 GB/s link and decimal MB. A 3 MB tile moves in 3e6 / 5e11 = 6e-6 s = 6 us.
Offload budget: T = T_launch + T_xfer + T_acc + T_return
large kernel: 2 us + 6 us + 4 us + 1 us = 13 us vs 20 us on CPU -> offload wins (20 / 13 = 1.54x)
small kernel: 2 us + 2 us + 1 us + 1 us = 6 us vs 5 us on CPU -> offload losesA prototype plan names the measurement that kills the pattern. Vague plans drift into schedule slips.
Example gates: kill the systolic candidate if median shapes fall below 60% utilization, or if spill traffic exceeds the stated bytes-per-operation budget.
Axis (1-5) | Dense 256x256 GEMM on systolic | Divergent loop on systolic |
|---|---|---|
Operation regularity | 5 | 2 |
Reuse under real shapes | 5 | 2 |
Divergence tolerance | 3 | 1 |
Compiler maturity | 4 | 2 |
1 / (0.2 + 0.8 / 5) = 1 / (0.2 + 0.16) = 1 / 0.36 = 2.78x.
The fixed cost. Batch more work per launch or keep the kernel on the CPU.