The covered hunt of lesson 6.9 catches all 24 planted bugs. It is also the most expensive testbench in the course: random stimulus, assertions, an arbitration checker, a coverage model and a phase written from its holes. A manager with half the budget asks: what is the cheapest mix of techniques that still catches everything?
Your task: choose the cheapest set of testbenches and proofs that catches all 24 DCRouter bugs, and say what that answer hides. Try it from the matrix in lesson 14.6 before reading on.
Two bugs, BUG_16 and BUG_23, were caught only by the covered hunt, by its phase written from coverage holes. That phase needs no random stimulus at all. Taken out as a directed bench of its own:
// uses: dcrouter
// Capstone 17.2: the register and port checks that the coverage model's holes asked for,
// written as a cheap directed bench with no random stimulus: reset value of CTRL, STATUS and
// an unmapped address, clear each counter and re-read all of them, saturate an error counter,
// and disable each port while offering it a packet.
module tb;
`include "dcr_env.svh"
initial begin
reset_dut();
check_reg(8'h00, 32'h3F, "CTRL after reset (R11)");
apb_rd(8'h04); // STATUS
apb_rd(8'h40); // unmapped (R12)
if (rd_err !== 1'b1) fail("R12: no pslverr for unmapped address 0x40");
for (int i = 0; i < 4; i++) queue_pkt(i, (i + 1) % 4, 0, 2); // one good packet per input
for (int i = 0; i < 4; i++) queue_pkt(i, i, 0, 2, 1); // one fault per input
wait_drain();
for (int k = 0; k < 4; k++) begin // clear each counter; only that one clears (R9, R10)
apb_wr(8'h10 + 8'(4 * k), 0); bad_pkts[k] = 0;
apb_wr(8'h20 + 8'(4 * k), 0); got_pkts[k] = 0;
for (int j = 0; j < 4; j++) begin
check_reg(8'h10 + 8'(4 * j), bad_pkts[j], $sformatf("ERR_CNT%0d after clearing ERR_CNT%0d", j, k));
check_reg(8'h20 + 8'(4 * j), got_pkts[j], $sformatf("PKT_CNT%0d after clearing PKT_CNT%0d", j, k));
end
end
for (int n = 0; n < 260; n++) queue_pkt(2, 3, 0, 1, 1); // saturate ERR_CNT2 (R9)
wait_drain(20000);
check_reg(8'h18, 255, "ERR_CNT2 after 260 faults");
for (int k = 0; k < 4; k++) begin // a disabled port accepts nothing (R1)
apb_wr(8'h00, 32'h30 | (32'hF & ~(32'h1 << k)));
queue_pkt(k, (k + 1) % 4, 0, 2);
repeat (30) @(posedge clk);
if (pending() != 4) fail($sformatf("R1: disabled port %0d accepted flits", k));
apb_wr(8'h00, 32'h3F);
wait_drain();
end
finish("targeted register and port bench");
end
endmoduleRun against the Bug Zoo it catches:
clean design: PASS
BUG_1 caught ERROR @1805: out3: unexpected flit 005d
BUG_2 caught ERROR @255: out0: unexpected flit 0580
BUG_8 caught ERROR @155: out0: out_err 1 on flit 0580, expected 0
BUG_11 caught ERROR @465: PKT_CNT1 after clearing PKT_CNT0: read 0, expected 2
BUG_14 caught ERROR @495: ERR_CNT2 after clearing ERR_CNT0: read 2, expected 1
BUG_15 caught ERROR @11985: ERR_CNT2 after 260 faults: read 4, expected ff
BUG_16 caught ERROR @1485: PKT_CNT3 after clearing PKT_CNT3: read 2, expected 0
BUG_17 caught ERROR @55: CTRL after reset (R11): read 37, expected 3f
BUG_18 caught ERROR @115: R12: no pslverr for unmapped address 0x40
BUG_19 caught ERROR @435: ERR_CNT1 after clearing ERR_CNT0: read 0, expected 1
BUG_21 caught ERROR @63725: 4 flits still pending after 5000 cycles (lost or stuck)
BUG_23 caught ERROR @12315: R1: disabled port 0 accepted flits
BUG_24 caught ERROR @205: out0: unexpected flit c58113 of 24, including BUG_16 and BUG_23, for a fraction of the covered hunt's machinery. It misses everything that needs traffic patterns: arbitration, back-pressure, and sleep in the middle of traffic (BUG_20, 22).
Testbenches share parts, so cost is counted per component: a mix pays for each component once, however many of its benches use it. The numbers are illustrative engineer-days; what matters is the reasoning, which the page recomputes from them.
Component | Effort (days) |
|---|---|
Shared environment: drivers, sinks, scoreboard, APB tasks (lesson 4.1) | 4 |
Directed tests (lesson 4.1) | 2 |
Error injection (lesson 4.7) | 2 |
Back-pressure and timing variation (lesson 4.8) | 1 |
Constrained-random hunt (lesson 4.10) | 3 |
Pin-level assertions (lesson 5.3) | 2 |
Arbitration checker (lesson 5.10) | 3 |
Coverage model (lesson 6.9) | 3 |
Phase written from coverage holes (lesson 6.9) | 2 |
Designer formal properties and harness (lesson 7.9) | 4 |
Sign-off gap bench (lesson 14.6) | 2 |
Targeted register and port bench (this lesson) | 2 |
Testbench or proof | Bugs caught | Cost alone (days) |
|---|---|---|
Directed | 10 | 6 |
Error injection | 13 | 6 |
Back-pressure | 12 | 5 |
Random hunt | 15 | 7 |
SVA hunt | 15 | 9 |
Checked hunt | 18 | 12 |
Covered hunt | 24 | 17 |
Formal proof | 9 | 4 |
Gap bench | 6 | 6 |
Targeted bench | 13 | 6 |
With ten candidates there are 1,023 non-empty mixes; the page checks all of them for missed bugs and cost.
Mix | Missed | Cost (days) |
|---|---|---|
Covered hunt alone | none | 17 |
Checked hunt + Targeted bench | none | 14 |
Cheapest complete mix: Random hunt + Formal proof + Targeted bench | none | 13 |
Cheapest without formal: Checked hunt + Targeted bench | none | 14 |
Cheapest without the targeted bench: Covered hunt | none | 17 |
How to read it: the cheapest complete mix, Random hunt + Formal proof + Targeted bench, costs 13 days against 17 for the covered hunt (Error injection + Back-pressure + Formal proof + Targeted bench ties at the same cost; the search prefers fewer runs). Formal takes over the arbitration bugs (BUG_5, 6, 7) that otherwise need the arbitration checker; the targeted bench takes the register and port bugs; the random hunt takes the traffic bugs. Without formal, the cheapest is Checked hunt + Targeted bench at 14 days. Without the targeted bench, nothing cheaper than the covered hunt catches BUG_16 and BUG_23. Different illustrative costs would change the winner, but not the shape: a mix of techniques with different pools beats one testbench that does everything.
Hindsight: the targeted bench was written from the covered hunt's coverage holes. Without the coverage model, nobody would have known that counter clears and disabled ports were untested.
Planted bugs are not real bugs: the mix is cheapest for these 24. Unknown bugs need breadth, which random stimulus and coverage provide and directed tests do not (lesson 4.2).
Single points of failure: in the cheapest mix 14 of the 24 bugs are caught by exactly one run; a regression change that weakens one bench lets them escape (lesson 14.6).
Formal's scope: the proofs cover the FIFOs, arbiter and credits under their assumptions; if the assumptions are wrong, the arbitration bugs have no other net in this mix (lesson 7.4).
The capstone's real answer is therefore two answers: the cheapest mix for the bugs you know, and the investment (coverage, plan, independent checks) that finds the bugs you do not.
BUG_16 and BUG_23 are caught by no other run: no other bench clears every counter and re-reads them, or offers a packet to a disabled port.
It was chosen knowing where the bugs were; for unknown bugs you need the breadth of random stimulus and the coverage model that found the holes in the first place.
A targeted directed bench caught 13 of 24 bugs, including the two only the covered hunt had found.
Searching every mix found a cheaper complete combination than the testbench that catches everything.
The cheap mix was only possible because coverage showed where to look.