DCRouter's scoreboard compared packets bit for bit: a flit was right or wrong. An AI accelerator multiplies matrices in float32 or smaller formats, adding thousands of products in a tree because that is fast in hardware. Its reference model, written in Python, adds them in a loop in double precision. The two will almost never agree to the last bit, and both are correct.

How do you check an accelerator whose correct answers are not exact?

The obvious answer, and why it breaks

The obvious answer is to compare results with the golden model bit for bit, as with DCRouter. Floating-point addition rounds at every step, so it is not associative: the same numbers summed in a different order give a slightly different result (fact V15), and a reference model usually also computes in higher precision than the hardware. An exact check therefore fails correct hardware, and a team that keeps seeing such failures learns to ignore them, or loosens the check until it catches nothing.

Build it: exact versus scaled tolerance

An illustrative accelerator computes 64-element dot products in float32 with tree reduction; the golden model uses double precision in a loop. A buggy version drops the last product in one of every 256 dot products. Both are checked exactly and with a tolerance that scales with the size of the terms.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

"""AI accelerators: checking floating-point results against a golden model.

An accelerator computes 64-element dot products in float32, accumulating in a tree (pairs,
then pairs of pairs), as hardware often does. The golden model computes the same dot product
in double precision in a simple loop. Two checks compare 2,000 random dot products:
  exact       results must be bit-identical
  tolerance   |hw - golden| <= 1e-5 * (sum of |a_i * b_i|) + 1e-30, a relative bound that
              scales with the size of the terms, as float32 accumulation error does
The same checks then run on a buggy accelerator that drops the last product in 1 of every 256
dot products (an illustrative control bug).
"""
import random
import struct

rng = random.Random(17)
f32 = lambda x: struct.unpack("f", struct.pack("f", x))[0]

def hw_dot(a, b, bug=False, k=0):
    terms = [f32(f32(x) * f32(y)) for x, y in zip(a, b)]
    if bug and k % 256 == 255:
        terms[-1] = 0.0
    while len(terms) > 1:                                 # tree reduction, rounding each add to float32
        terms = [f32(terms[i] + terms[i + 1]) for i in range(0, len(terms), 2)]
    return terms[0]

def golden(a, b):
    return sum(f32(x) * f32(y) for x, y in zip(a, b))

vectors = [([rng.uniform(-1, 1) for _ in range(64)], [rng.uniform(-1, 1) for _ in range(64)]) for _ in range(2000)]
for name, bug in [("correct accelerator", False), ("buggy accelerator", True)]:
    exact_fail = tol_fail = 0
    for k, (a, b) in enumerate(vectors):
        h, g = hw_dot(a, b, bug, k), golden(a, b)
        scale = sum(abs(f32(x) * f32(y)) for x, y in zip(a, b))
        exact_fail += (h != g)
        tol_fail += abs(h - g) > 1e-5 * scale + 1e-30
    print(f"{name:20s} exact check fails {exact_fail:4d} of 2000; tolerance check fails {tol_fail:3d} of 2000")
    if not bug:
        ok_exact, ok_tol = exact_fail, tol_fail
    else:
        bug_tol = tol_fail
assert ok_exact > 1500 and ok_tol == 0 and bug_tol > 0
print("PASS: an exact check fails every correct result; a scaled tolerance passes them all and still catches the dropped products")

It prints:

correct accelerator  exact check fails 2000 of 2000; tolerance check fails   0 of 2000
buggy accelerator    exact check fails 2000 of 2000; tolerance check fails   7 of 2000
PASS: an exact check fails every correct result; a scaled tolerance passes them all and still catches the dropped products

How to read it: the exact check fails all 2,000 results of the correct accelerator, so it carries no information: correct and buggy designs look the same. The scaled tolerance passes every correct result and fails exactly the 7 dot products where a product was dropped. The scale matters: an absolute tolerance would be too loose for small results and too tight for large ones, because rounding error grows with the magnitude of the terms being added.

The idea: layered oracles

A useful second reference is a bit-accurate model that reproduces the hardware's exact order and formats: then the RTL can be compared bit for bit again, and the separate question of whether that order is accurate enough is answered once, against the high-precision model.

What else changes

• The input space is data: random values miss the distributions of real weights and activations; include real tensors, extreme magnitudes, zeros, denormals and NaNs.

• Control is most of the bug risk: tiling, addressing, synchronisation and scheduling between engines; these produce dropped or repeated data that a tolerance must be tight enough to see.

• Compiler and hardware are verified together: the same network compiled differently exercises different schedules.

• Workloads are long: whole networks run on emulation (Part 10).

How real teams verify accelerators

What breaks

A team (illustrative) set its tolerance by raising it until a failing regression passed. The failures had been caused by an addressing bug that occasionally read a neighbouring tile's data, whose values were close in magnitude; the loosened tolerance absorbed it. A tolerance should be derived from the formats and accumulation length, then validated by planting bugs, as in the model above, before it is trusted.

Key takeaways

• Correct floating-point hardware differs from a golden model; exact checks fail it.

• Use tolerances derived from the arithmetic and scaled to the terms, validated with planted bugs.

• Layer oracles: bit-exact units, tolerant kernels, network accuracy.

Related topics

• Part 15 overview: Verification in Context