Halfway through closure, a regression has hit 28 of DCRouter's 32 header bins. The four holes are all length-0 headers at particular destinations and priorities, each about one packet in 400 under the random distribution. Every new test spends ninety-nine packets out of a hundred re-hitting bins that are already full. The coverage report knows exactly which four bins are missing; the stimulus generator does not read it.

How can coverage feedback make closure faster?

The obvious answer, and why it breaks

The obvious fixes are more tests or a directed test per hole. More tests pay at the coupon collector's slow rate (lesson 4.2). A directed test per hole works for a handful of holes, but holes keep appearing as the model grows, seeds change and the design changes, and hand-writing a test for each does not scale. The information needed to steer is already in the coverage database.

The idea: close the loop

• Read the holes after each test or batch.

• Turn a hole into constraints: a bin (destination 2, priority 1, length 0) becomes inline constraints for some of the next packets.

• Keep part of each test blind, so breadth and unexpected combinations continue.

• Repeat until the reachable bins are closed; what remains goes to triage (lesson 6.6).

Build it: blind against driven

Both generators send 20 packets per test until all 32 header bins are hit. The driven one sends 10 packets at a randomly chosen hole and 10 blind packets each test. Each is run 400 times.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

"""Coverage-driven stimulus against blind random, on DCRouter's 32 header bins.

The coverage model is lesson 6.9's header cross: 4 destinations x 2 priorities x 4 length
classes (len0, len1, len2-14, len15) = 32 bins. Each test sends 20 packets.
  blind    every packet from lesson 4.10's distribution: len0 2%, len1 10%, len2-14 70%,
           len15 18%; priority and destination uniform
  driven   after each test, look at the holes; the next test sends 10 packets constrained to
           one randomly chosen hole and 10 blind packets
Both run until all 32 bins are hit; 400 runs each. The model prints average coverage after
each number of tests and the average number of tests to close.
"""
import random

LEN_W = [("len0", 0.02), ("len1", 0.10), ("len2-14", 0.70), ("len15", 0.18)]
BINS = [(d, p, l) for d in range(4) for p in range(2) for l, _ in LEN_W]

def blind_packet(rng):
    u, acc = rng.random(), 0.0
    for l, w in LEN_W:
        acc += w
        if u < acc:
            break
    return (rng.randrange(4), rng.randrange(2), l)

def run(driven, rng, checkpoints):
    seen, t, curve = set(), 0, {}
    target = None
    while len(seen) < len(BINS):
        t += 1
        for k in range(20):
            pkt = target if (driven and target and k < 10) else blind_packet(rng)
            seen.add(pkt)
        if t in checkpoints:
            curve[t] = len(seen)
        holes = [b for b in BINS if b not in seen]
        target = rng.choice(holes) if holes else None
    for c in checkpoints:
        curve.setdefault(c, len(BINS))
    return t, curve

CK = [1, 2, 5, 10, 20, 40, 80]
rng = random.Random(9)
results = {}
for name, driven in [("blind", False), ("driven", True)]:
    tests, curves = [], []
    for _ in range(400):
        t, c = run(driven, rng, CK)
        tests.append(t)
        curves.append(c)
    avg = [sum(c[k] for c in curves) / len(curves) / len(BINS) * 100 for k in CK]
    results[name] = (sum(tests) / len(tests), max(tests), avg)

print("tests:            " + " ".join(f"{k:>6d}" for k in CK))
for name, (mean, worst, avg) in results.items():
    print(f"{name:6s} coverage % " + " ".join(f"{a:6.1f}" for a in avg) + f"   tests to close: mean {mean:.1f}, worst {worst}")
speed = results["blind"][0] / results["driven"][0]
print(f"coverage-driven closes {speed:.1f} times faster on average")

assert results["driven"][0] < results["blind"][0] / 2
print("PASS: steering half of each test at a hole closes the header cross in less than half the tests")

It prints:

tests:                 1      2      5     10     20     40     80
blind  coverage %   36.7   51.5   70.7   82.3   90.4   96.6   99.6   tests to close: mean 54.9, worst 180
driven coverage %   36.8   47.9   69.7   91.9  100.0  100.0  100.0   tests to close: mean 12.4, worst 17
coverage-driven closes 4.4 times faster on average
PASS: steering half of each test at a hole closes the header cross in less than half the tests

How to read it: blind random needs 54.9 tests on average, and 180 in its worst run, because the last length-0 bins arrive by luck. Driven closes in 12.4 on average and never more than 17. Note the early cost: after two tests driven is slightly behind (47.9% against 51.5%), because half its packets went to a single hole. Steering trades breadth for focus, which pays off once only holes remain.

Ways to steer

Cautions

• Hitting a bin is not testing it: steered stimulus still needs active checks (lesson 6.1).

• Keep blind stimulus in the mix: steering only toward the model finds only what the model names.

• Event and sequence bins are harder to steer than value bins; they often need scenarios (lesson 4.6).

How real teams use coverage-driven stimulus

What breaks

A team (illustrative) steered all of its stimulus toward coverage holes and closed its model quickly. The model happened to omit back-pressure on one interface, and the steered tests, now narrow, never produced it; earlier blind tests had produced it occasionally. A bug there escaped. Keeping half of each test blind, as the model above does, preserves the breadth that finds what the model forgot.

Key takeaways

• Coverage-driven stimulus turns holes into constraints for the next tests.

• On DCRouter's header cross it closed 4.4 times faster on average and capped the worst case at 17 tests.

• Keep a blind share for breadth, send stubborn holes to directed tests or formal, and keep checks active.

Related topics

• Part 6 overview: Coverage