Each test writes a coverage database. Closure is decided on the merge of many of them. Three operations happen on that merge: union the bins, rank the tests by what they add, and list the holes. Every simulator vendor ships tools for this, such as Questa vcover merge and vcover ranktest, or VCS urg with its grading option; check your tool's manual for the exact switches.
Step | Rule | Why |
|---|---|---|
Merge | Union of bins across passing tests only | Coverage from a failing test was not checked, so it earns no credit. |
Version | Merge only databases built from the same coverage model | If bins changed between runs, tools handle the mismatch differently. Treat a model change as a new baseline. |
Rank | Greedy: take the test that adds most new bins, repeat | Finding the smallest covering set is NP-hard; greedy is fast and within a log factor of optimal. |
Holes | Every uncovered bin gets an owner and an action | Directed test, bias change, waiver with reason, or proof of unreachability. |
The bench below models a regression over the 34 legal bins of the burst cross from Coverage Filters. Each "test" is 20 bursts from a weighted generator: length 1, 2, 4, 8, 16 with weights 40, 25, 20, 10, 5 percent; beat size 1, 2, 4, 8 bytes with 10, 20, 40, 30; WRAP 20 percent. Illegal draws are re-drawn, as a constraint solver would. A fixed xorshift generator replaces randomize() so every simulator produces the same numbers.
// Regression model: 50 seeds x 20 constrained-random bursts over the 34 legal
// bins of the len x size x burst cross. Merge, closure curve, greedy ranking.
module tb_closure;
localparam int SEEDS = 50, TXNS = 20, LEGAL = 34;
bit [39:0] seed_mask [SEEDS]; // bins each seed hit (bit = cross index)
int errors = 0;
// Legal-bin test: same filter as the coverage-filters lesson.
function automatic bit legal(int l, int s, int w);
return ((l << s) <= 64) && !(w == 1 && l == 1);
endfunction
function automatic int idx(int l, int s, int w);
return $clog2(l) * 8 + s * 2 + w;
endfunction
// xorshift32: same sequence on every simulator.
function automatic bit [31:0] next(ref bit [31:0] st);
st ^= st << 13; st ^= st >> 17; st ^= st << 5;
return st;
endfunction
// One test: TXNS bursts drawn with the generator's weights, illegal draws re-drawn.
function automatic bit [39:0] run_test(int seed);
bit [31:0] st;
bit [39:0] m;
st = 32'(seed) * 32'h9E3779B9 + 32'd1;
m = '0;
for (int t = 0; t < TXNS; t++) begin
int l, s, w, r;
do begin
r = int'(next(st) % 100); // len: 1 40%, 2 25%, 4 20%, 8 10%, 16 5%
l = r < 40 ? 1 : r < 65 ? 2 : r < 85 ? 4 : r < 95 ? 8 : 16;
r = int'(next(st) % 100); // size: 0 10%, 1 20%, 2 40%, 3 30%
s = r < 10 ? 0 : r < 30 ? 1 : r < 70 ? 2 : 3;
w = (next(st) % 100 < 20) ? 1 : 0; // WRAP 20%
end while (!legal(l, s, w));
m[idx(l, s, w)] = 1'b1;
end
return m;
endfunction
initial begin
bit [39:0] merged, ranked;
int curve [4];
int order [$];
int gains [$];
bit used [SEEDS];
merged = 0; ranked = 0; curve = '{5, 10, 20, 50};
foreach (seed_mask[i]) seed_mask[i] = run_test(i + 1);
// Merge: a bin is covered if any seed hit it.
foreach (seed_mask[i]) begin
merged |= seed_mask[i];
foreach (curve[k]) if (i + 1 == curve[k])
$display("after %0d seeds: %0d of %0d bins", i + 1, $countones(merged), LEGAL);
end
// Greedy ranking: take the seed that adds most new bins; stop when none adds.
for (int step = 0; step < SEEDS; step++) begin
int best, best_gain;
best = -1; best_gain = 0;
foreach (seed_mask[i]) begin
int g;
g = $countones(seed_mask[i] & ~ranked);
if (!used[i] && g > best_gain) begin best = i; best_gain = g; end
end
if (best < 0) break;
used[best] = 1'b1;
ranked |= seed_mask[best];
order.push_back(best + 1);
gains.push_back(best_gain);
end
foreach (order[i]) $display("rank %0d: seed %0d, new bins %0d", i + 1, order[i], gains[i]);
// Holes: legal bins no seed hit.
for (int l = 1; l <= 16; l *= 2)
for (int s = 0; s < 4; s++)
for (int w = 0; w < 2; w++)
if (legal(l, s, w) && !merged[idx(l, s, w)])
$display("hole: len %0d, %0d-byte beats, %s", l, 1 << s, w != 0 ? "WRAP" : "INCR");
// Checks that hold for any data: the ranked set reproduces the merge,
// every ranked seed adds something, and greedy gains never increase.
if (ranked != merged) errors++;
// Data-dependent results, recomputed independently from the same generator.
if ($countones(merged) != 33 || order.size() != 8) errors++;
foreach (gains[i]) begin
if (gains[i] == 0) errors++;
if (i > 0 && gains[i] > gains[i - 1]) errors++;
end
if (errors == 0) $display("PASS: %0d seeds merged to %0d of %0d bins; %0d ranked seeds keep all of it",
SEEDS, $countones(merged), LEGAL, order.size());
else $fatal(1, "FAIL: %0d merge or ranking errors", errors);
$finish;
end
endmoduleThe run prints the closure curve, the ranking and the hole. These are the measured results, and a separate Python model of the same generator reproduced every number:
Seeds | Bins covered (measured) | Expected from bin probabilities |
|---|---|---|
5 | 25 of 34 | 24.4 |
10 | 28 of 34 | 28.6 |
20 | 32 of 34 | 31.5 |
50 | 33 of 34 | 33.4 |
The expected column is the sum over bins of the chance each was hit at least once, computed from the weights. Measured and expected agree, which is a useful sanity check on a real regression too: a curve far below its expectation means the generator is not producing the distribution you think.
Ranking keeps all 33 bins with 8 of the 50 seeds: seeds 9, 16, 3, 50, 14, 1, 15 and 20, adding 15, 7, 3, 3, 2, 1, 1 and 1 bins. The remaining 42 seeds add no coverage. They may still find bugs, which is why ranking chooses a quick regression, not a replacement for random runs.
The bench checks properties that hold for any data: the ranked set reproduces the merge, every ranked seed adds a bin, and greedy gains never increase. It also checks the two data-dependent results, 33 bins and 8 ranked seeds. A ranking loop that forgets to update its covered set picks all 50 seeds and fails. A merge that keeps only the last seed's bins no longer matches the ranked union and fails.
The one hole is WRAP, length 16, 1-byte beats. Its chance per legal burst is 0.05 x 0.10 x 0.20 / 0.905 = 0.0011, where 0.905 is the probability that a draw is legal. Per test of T bursts, and over N tests:
With p = 0.0011 and T = 20, p_test = 0.0219. Fifty seeds hit it with probability 1 - 0.978^50 = 0.67, so missing it here was not bad luck: it happens in one regression in three. Reaching 95% confidence (c = 0.95) takes N = 136 seeds. The second rarest bin (p = 0.0022) needs 68. One directed test closes the rarest bin, which costs a fraction of one seed.
Compute p for the rarest bins from the generator's weights before planning the regression. The arithmetic takes minutes; the compute it saves is hours.
If N is large, change the distribution (a bias test) or write a directed test. More seeds is the most expensive way to close a rare bin.
If p is zero the bin is unreachable with this generator. No seed budget closes it.
Every plan row has its coverage items at goal, or a recorded decision (waiver with reason, or proof of unreachability).
The closure curve is flat: new seeds add no bins, and the remaining holes are understood, not just listed.
The checks were shown to work: planted bugs fail the regression. Coverage proves stimulus reached a state, not that a checker looked at it.
The bug-find rate has dropped and stayed low across recent regressions, with no open critical bugs.
Sign-off metrics covers the review itself. At scale the trap is treating the number as the goal: 100% of a weak model with silent checkers is worse than 92% of a strong one with every hole explained.
p_test = 1 - 0.998^50 = 0.095. N = ln(0.05) / ln(0.905) = 30 seeds, rounded up. A directed test is still cheaper.
Ranking optimises coverage, not bug finding. The other seeds exercise different orderings within the same bins and can still find bugs, and after any testbench change the ranking is stale. Use the ranked set as a quick regression, and keep running random seeds.