A configurable interconnect IP (illustrative) ships to its fifth customer, the first to use two ports with 32-byte data. Every regression had run the default configuration: four ports, 16 bytes. In the customer's configuration, a width conversion in the arbiter truncates every other burst. The bug had been in the RTL since the first release, in a combination nobody had built.

How do configurations and modes hide bugs, and how many of them must be verified?

The obvious answer, and why it breaks

The obvious approach verifies the default configuration thoroughly, on the reasoning that the logic is the same in every configuration. It is not: parameters change widths, generate different logic, change which branches are reachable and which limits are hit. Registers and modes do the same at run time. A bug in a branch that only one combination reaches is invisible in every other, and the default is only one point in the space.

The idea: the space multiplies

• Build-time parameters: each value can generate different logic; each combination is effectively a different design.

• Run-time registers and modes: DCRouter's CTRL alone has 64 settings (4 port enables, priority on or off, low power on or off), before any traffic.

• Interactions matter more than single values: a bug often needs two settings together (two ports and wide data; priority off and a heavy load).

Build it: covering pairs, not products

If most configuration bugs need at most two settings together, you do not need every combination; you need every pair of values to appear in some build. The model compares all 256 builds of a four-parameter IP with 16 builds chosen from orthogonal Latin squares (fact M5), and with 16 random builds.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

"""Configuration bugs: covering every pair of settings without testing every combination.

An IP block with four parameters, each with four legal values (illustrative: FIFO depth,
data width, number of ports, arbitration mode). Testing every combination needs 4^4 = 256
builds. A configuration bug that needs two specific values together (say depth 2 with
4 ports) is found by any set of builds that contains that pair.
Two mutually orthogonal Latin squares of order 4 give 16 builds in which every pair of
values of every two parameters appears at least once. The model builds them, checks the
pair coverage, and compares with 16 builds chosen at random.
"""
import itertools, random

V = 4
# Latin squares over GF(4) elements {0,1,2,3}: L1(r,c) = r + c, L2(r,c) = r + 2c (field arithmetic)
EXP = [1, 2, 3, 1, 2, 3]                  # powers of the GF(4) generator: 1, a, a+1
LOG = {1: 0, 2: 1, 3: 2}
def gmul(a, b):
    return 0 if a == 0 or b == 0 else EXP[LOG[a] + LOG[b]]
builds = [(r, c, r ^ c, r ^ gmul(2, c)) for r in range(V) for c in range(V)]   # addition in GF(4) is XOR

def pair_coverage(rows):
    have = set()
    for row in rows:
        for (i, a), (j, b) in itertools.combinations(enumerate(row), 2):
            have.add((i, j, a, b))
    total = len(list(itertools.combinations(range(4), 2))) * V * V
    return len(have), total

full = list(itertools.product(range(V), repeat=4))
c16, total = pair_coverage(builds)
rng = random.Random(4)
rand_cov = sorted(pair_coverage(rng.sample(full, 16))[0] for _ in range(200))
print(f"every combination: {len(full)} builds")
print(f"orthogonal array: {len(builds)} builds cover {c16} of {total} value pairs")
print(f"16 random builds: cover {rand_cov[len(rand_cov) // 2]} of {total} pairs (median of 200 tries), worst {rand_cov[0]}")
assert c16 == total == 96 and rand_cov[-1] < total
print("PASS: 16 well-chosen builds cover all 96 value pairs that 256 builds would; 16 random builds never do")

It prints:

every combination: 256 builds
orthogonal array: 16 builds cover 96 of 96 value pairs
16 random builds: cover 63 of 96 pairs (median of 200 tries), worst 48
PASS: 16 well-chosen builds cover all 96 value pairs that 256 builds would; 16 random builds never do

How to read it: 16 well-chosen builds contain all 96 pairs of values that the full 256 would, so any bug triggered by two settings together appears in one of them. Sixteen random builds typically cover only 63 pairs and never all 96. NIST studies of real failures found most were triggered by one or two interacting parameters and all by six or fewer (fact M5), which is why pairwise (and higher-strength) selection is effective.

The design space

How real teams verify configurable designs

What breaks

DCRouter has four configuration bugs. Two concern the priority setting: one ignores priority when it is enabled (BUG_6), the other applies it when it is disabled (BUG_7). A test that never writes CTRL runs only the reset configuration, with priority enabled, so it can find BUG_6 but never BUG_7, which exists only in the configuration nobody set. Another lets a disabled port accept data (BUG_23): invisible unless a test disables a port.

Key takeaways

• Parameters, registers and modes multiply the space; the default configuration is one point in it.

• Most configuration bugs need one or two settings together, so cover every pair, not every combination.

• Randomise run-time setup, measure configuration coverage, and test every shipped configuration.

Related topics

• Part 1 overview: What Can Go Wrong