The previous lesson defined the rows CV1 to CV9 and the mirror that counts them. A row is only closed when its goal is met across the regression, so this lesson merges many short runs, ranks them, and reads what the leftover holes say about the stimulus. The row names below refer to that table.
The full bench (300 items per port per phase, then a directed empty-table step) meets every goal in one run on every seed we tried: 1 to 30 (11 is the default) and 99. To see closure work, the bench was run with +items=20 +empty_reads=0 (the directed step off, as the bench was first written) and seeds 1 to 10, 104 to 116 cycles each. No single run met every goal; each left 4 to 8 rows short. The table is the measured mirror output; the bench below merges it.
// Closure over 10 short capstone runs (+items=20 +empty_reads=0, +seed=1..10). The hit
// counts are the coverage-mirror rows those runs printed; this bench merges them.
module tb_capstone_closure;
localparam int SEEDS = 10, ROWS = 21;
string row_name [ROWS] = '{
"CV1 depth 0", "CV1 depth 1-3", "CV1 depth 4-7", "CV1 depth 8+ (full)", "CV1b entry 7",
"CV2 same-ID depth >= 2", "CV3 cross-ID overtake",
"CV4 p0 lane 0", "CV4 p0 lane 1", "CV4 p0 lane 2", "CV4 p0 lane 3",
"CV4 p1 lane 0", "CV4 p1 lane 1", "CV4 p1 lane 2", "CV4 p1 lane 3",
"CV5 read-after-write", "CV6 stall >= 3", "CV7 p0 wins", "CV7 p1 wins",
"CV8 reset busy", "CV9 reuse after reset"};
int goal [ROWS] = '{5, 10, 10, 3, 3, 10, 10, 5, 5, 5, 5, 5, 5, 5, 5, 10, 5, 5, 5, 1, 1};
int hits [SEEDS][ROWS] = '{
'{2, 25, 19, 1, 1, 6, 11, 12, 11, 8, 6, 11, 12, 7, 7, 5, 2, 42, 44, 1, 4},
'{2, 29, 18, 0, 0, 6, 9, 10, 10, 4, 7, 14, 10, 14, 9, 6, 0, 44, 46, 1, 3},
'{2, 14, 40, 3, 3, 7, 17, 10, 9, 6, 5, 9, 7, 4, 4, 11, 4, 44, 46, 1, 3},
'{2, 22, 30, 1, 1, 9, 13, 10, 11, 12, 6, 14, 7, 5, 6, 10, 3, 44, 46, 1, 3},
'{2, 19, 33, 4, 2, 5, 11, 14, 7, 5, 4, 9, 7, 4, 3, 3, 1, 43, 44, 1, 4},
'{2, 22, 23, 2, 2, 9, 15, 14, 14, 9, 7, 8, 12, 6, 8, 7, 3, 44, 46, 1, 3},
'{3, 17, 34, 0, 2, 8, 16, 13, 7, 7, 5, 10, 12, 9, 10, 3, 2, 44, 46, 1, 3},
'{2, 19, 29, 3, 2,14, 13, 12, 7, 6, 4, 8, 11, 4, 6, 6, 3, 42, 45, 1, 2},
'{2, 21, 26, 1, 1, 4, 10, 12, 8, 12, 8, 13, 11, 8, 7, 6, 6, 43, 45, 1, 4},
'{3, 22, 20, 4, 3, 4, 17, 8, 8, 2, 9, 9, 10, 9, 11, 12, 3, 42, 44, 1, 4}};
// Rows below goal for one set of counts.
function automatic int holes(int h [ROWS]);
int n = 0;
for (int r = 0; r < ROWS; r++) if (h[r] < goal[r]) n++;
return n;
endfunction
// Total shortfall: how many hits are still missing across all rows.
function automatic int deficit(int h [ROWS]);
int d = 0;
for (int r = 0; r < ROWS; r++) if (h[r] < goal[r]) d += goal[r] - h[r];
return d;
endfunction
initial begin
int per_seed [SEEDS], curve [SEEDS], merged [ROWS], ranked [ROWS], order [$];
int errors = 0;
bit used [SEEDS];
merged = '{default: 0}; ranked = '{default: 0};
// 1. Each run on its own.
for (int s = 0; s < SEEDS; s++) per_seed[s] = holes(hits[s]);
$display("holes per run: %p", per_seed);
// 2. Rows that no single run closes: a stimulus question, not a seed question.
for (int r = 0; r < ROWS; r++) begin
bit always_hole = 1;
for (int s = 0; s < SEEDS; s++) if (hits[s][r] >= goal[r]) always_hole = 0;
if (always_hole) $display("below goal in every run: %s", row_name[r]);
if (always_hole != (r == 0)) errors++;
end
// 3. Merge counts run by run (what a coverage merge does with bin hits).
for (int s = 0; s < SEEDS; s++) begin
for (int r = 0; r < ROWS; r++) merged[r] += hits[s][r];
curve[s] = holes(merged);
end
$display("rows below goal after merging 1..10 runs: %p", curve);
// 4. Greedy ranking: take the run that removes most of the shortfall.
while (deficit(ranked) > 0) begin
int best = -1, best_d = 0;
for (int s = 0; s < SEEDS; s++) begin
int t [ROWS], d;
if (used[s]) continue;
for (int r = 0; r < ROWS; r++) t[r] = ranked[r] + hits[s][r];
d = deficit(t);
if (best < 0 || d < best_d) begin best = s; best_d = d; end
end
used[best] = 1;
for (int r = 0; r < ROWS; r++) ranked[r] += hits[best][r];
order.push_back(best + 1);
$display("rank %0d: seed %0d, shortfall left %0d", order.size(), best + 1, best_d);
end
// Results recomputed independently (Python) from the same table.
if (per_seed != '{6, 8, 5, 5, 8, 6, 6, 6, 5, 4}) errors++;
if (curve[0] != 6 || curve[1] != 4 || curve[2] != 0 || curve[9] != 0) errors++;
if (order.size() != 2 || order[0] != 3 || order[1] != 7) errors++;
if (errors == 0)
$display("PASS: no single run closes; 3 merged runs close all %0d rows; seeds 3 and 7 alone close them", ROWS);
else $fatal(1, "FAIL: %0d closure results differ", errors);
$finish;
end
endmoduleThe bench passes and prints the closure curve: 6 rows short after one run, 4 after two, none after three. Ranking finds that seeds 3 and 7 together close every row. One row, CV1 depth 0, is below goal in all ten runs, with 2 or 3 hits each.
Mutant run. Merging pass/fail flags instead of hit counts, so a row counts as closed only if some single run met its goal, gives a curve that ends at 1 row short after ten runs, and the bench fails. CV1 depth 0 never closes under that rule.
Which merge is right depends on what the row means. "The table was found empty at least five times across the regression" is a count, and counts add. "Every test drains the table at least once" is a per-test property, and only flags express it. Decide per row and write it in the plan; the two answers differ here.
CV1 depth 0 is the more useful finding. It is short in every run because the stimulus almost never lets the table empty: drivers send back to back, and gaps are short. More seeds will not fix a structural gap. The full-length run reached the goal on most seeds only because it is longer: without the directed step, 6 of 31 seeds (seeds 1 to 30 and 99; failing: 4, 7, 8, 10, 26 and 99) ended with 3 or 4 hits against a goal of 5, so the gate would fail about one correct run in five. The fix is the directed scenario now in the bench as phase D: wait for the table to empty, send one read, five times. With it, all 31 seeds pass, with 8 to 18 hits on the row. The arithmetic for when more seeds will help is in Closure at Scale.
Holes can also be the only sign of a bug. Two experiments with the full bench:
Experiment | Checkers | Coverage |
|---|---|---|
| None fired; the bug is never exercised. Only the bench's own check that a reset happened fails | CV8 and CV9 at 0: holes |
| None fired; port 1 is still served when port 0 is idle | CV7 port 1 wins at 0: hole |
In both cases a reviewer who waived the hole as "rare" would have signed off a broken design. A hole on a row that the plan ranked high needs a reason, not a waiver.
The one the plan row specifies. If the row only needs the empty-table case to be exercised a few times in the regression, the count merge is correct and the row is closed. If it means every test must drain the table, the row is open, and the fix is stimulus, not seeds.