Tensor Cores & Sparse Compute: Tricky Q&A

[INT][ACCEL][TENSOR-CORES-SPARSE]

Q: Why can tensor-core peak FLOPs be far above delivered throughput in real workloads?

A:
Peak FLOPs assume ideal tile occupancy and uninterrupted data supply, but real kernels often stall on memory movement, synchronization, or shape misalignment. If shared-memory staging, register reuse, and warp scheduling are not tuned, tensor pipelines idle even when the hardware supports much higher compute. Delivered throughput is therefore constrained by dataflow quality, not just arithmetic capability.

FOLLOW-UP TRAP: Treating low tensor-core throughput as purely a hardware limit instead of a dataflow issue.

[INT][ACCEL][TENSOR-CORES-SPARSE]

Q: How does mixed-precision matmul improve speed without always hurting model quality?

A:
Lower-precision math increases tensor-core throughput and cuts bandwidth, while wider accumulation and selective precision controls preserve numerical stability. Training can use scaling and precision overrides for sensitive layers, and inference can rely on calibration and outlier handling to maintain accuracy. The approach works when precision choices are operator-aware rather than globally forced.

FOLLOW-UP TRAP: Assuming one low-precision format should be applied uniformly to every operator.

[INT][ACCEL][TENSOR-CORES-SPARSE]

Q: What makes structured sparsity easier to accelerate than unstructured sparsity?

A:
Structured patterns constrain where zeros appear, so hardware and compilers can skip work with predictable scheduling and minimal control overhead. This regularity preserves parallel efficiency and memory coalescing, enabling practical speedups on supported tensor-core paths. Unstructured sparsity is more flexible but introduces irregular indexing and execution paths that are harder to optimize.

FOLLOW-UP TRAP: Assuming any sparsity pattern automatically maps to fast sparse tensor-core execution.

[INT][ACCEL][TENSOR-CORES-SPARSE]

Q: When is unstructured sparsity still worth considering despite runtime overhead?

A:
It is valuable when high compression ratios materially reduce memory footprint or bandwidth and when quality targets are easier to maintain with unconstrained pruning. It can also help deployment scenarios constrained by model size more than raw throughput. Teams should choose it only after benchmarking end-to-end latency, energy, and fallback behavior against structured alternatives.

FOLLOW-UP TRAP: Choosing unstructured sparsity from pruning percentage alone without system-level benchmarks.

SYMPTOM -> BOTTLENECK CLASS -> ARTIFACT -> OWNER ACTION -> VALIDATION

SPARSE TENSOR EXECUTION

model graph -> compiler lower -> sparse or dense kernel path -> runtime scheduling -> SLA outcome