A new engineer's first testbench drives a FIFO with pushes and pops and prints every value that comes out. The waveform looks plausible, the printout looks plausible, and the test "passes" because nothing crashed. Run the same testbench on a FIFO with a broken full flag and it prints different numbers that look just as plausible. Nobody can tell from the output which run was correct, least of all a regression script running ten thousand tests overnight.
What turns a testbench that runs into a testbench that checks?
The obvious answer is that someone looks at the waveform or the log. That works once, for a small test, with an engineer who knows the right answers. It does not scale to thousands of tests, millions of cycles and every overnight run, and people miss things they are not looking for. A test whose verdict depends on a person reading its output is a demonstration, not a test.
Part | Job | Without it |
|---|---|---|
Reference (oracle) | Knows the right answer for every output | Nothing to compare against (lesson 2.1) |
Comparison on every output | Turns each mismatch into a counted error | Wrong values pass silently |
End-of-test checks | Expected outputs that never arrived; counters; leftovers | Lost data passes (lesson 5.7) |
Watchdog | Ends a hung run with a failure | A hang looks like a slow pass, or blocks the regression |
One verdict line | PASS or FAIL that a script can read | Results need a person to interpret |
Evidence the checker can fail | Shows the checks are real | A checker that checks nothing passes everything (lesson 5.8) |
Two copies of a 4-entry FIFO receive identical random traffic: one correct, one whose full flag rises one entry late. Each copy gets the same checker: a reference queue predicts every popped word, mismatches are counted, leftovers are checked at the end, and a watchdog stops a hung run. The test passes only if the checker is silent on the correct FIFO and reports the buggy one.
// Lesson 5.1: a self-checking testbench. Two copies of a 4-entry FIFO run the same random
// pushes and pops: one correct, one whose full flag rises one entry late (it accepts a fifth
// word and overwrites the oldest). Each copy has the same checker: a reference queue predicts
// every popped word, mismatches are counted, leftovers are checked at the end, and a watchdog
// ends a hung run. The test passes only if the checker is silent on the correct FIFO and
// reports the buggy one, so the run also shows the checker can fail.
module fifo4 #(parameter bit BUGGY = 0) (
input logic clk, rst, push, pop,
input logic [7:0] din,
output logic [7:0] dout,
output logic full, empty
);
logic [7:0] mem [0:3];
logic [1:0] wp, rp;
logic [2:0] count;
assign full = BUGGY ? (count > 3'd4) : (count == 3'd4);
assign empty = (count == 3'd0);
assign dout = mem[rp];
always_ff @(posedge clk)
if (rst) begin wp <= 0; rp <= 0; count <= 0; end
else begin
if (push && !full) begin mem[wp] <= din; wp <= wp + 2'd1; end
if (pop && !empty) rp <= rp + 2'd1;
count <= count + 3'(push && !full) - 3'(pop && !empty);
end
endmodule
module fifo_checker (input logic clk, rst, push, pop, full, empty, input logic [7:0] din, dout);
logic [7:0] model [$];
int errors = 0, checked = 0;
always @(posedge clk) if (!rst) begin
if (pop && !empty) begin // every popped word must be the oldest pushed
if (model.size() == 0) errors++;
else begin
if (dout !== model[0]) errors++;
void'(model.pop_front());
end
checked++;
end
if (push && !full) model.push_back(din);
end
function automatic int leftovers(); // end of test: words the model still holds
return model.size();
endfunction
endmodule
module tb;
logic clk = 0, rst = 1, push = 0, pop = 0;
always #5 clk = ~clk;
logic [7:0] din = 0, dout_g, dout_b;
logic full_g, empty_g, full_b, empty_b;
fifo4 #(0) good (.clk, .rst, .push, .pop, .din, .dout(dout_g), .full(full_g), .empty(empty_g));
fifo4 #(1) bad (.clk, .rst, .push, .pop, .din, .dout(dout_b), .full(full_b), .empty(empty_b));
fifo_checker chk_g (.clk, .rst, .push, .pop, .full(full_g), .empty(empty_g), .din, .dout(dout_g));
fifo_checker chk_b (.clk, .rst, .push, .pop, .full(full_b), .empty(empty_b), .din, .dout(dout_b));
initial begin
repeat (2) @(posedge clk); rst <= 0;
for (int k = 0; k < 400; k++) begin // random traffic, biased to fill the FIFO
@(posedge clk);
push <= ($urandom_range(0, 99) < 60);
pop <= ($urandom_range(0, 99) < 40);
din <= 8'($urandom);
end
@(posedge clk); push <= 0; pop <= 1; // drain
repeat (8) @(posedge clk); pop <= 0;
@(posedge clk);
$display("correct FIFO: %0d pops checked, %0d errors, %0d leftovers", chk_g.checked, chk_g.errors, chk_g.leftovers());
$display("buggy FIFO: %0d pops checked, %0d errors, %0d leftovers", chk_b.checked, chk_b.errors, chk_b.leftovers());
if (chk_g.errors == 0 && chk_g.leftovers() == 0 && chk_g.checked > 100 && chk_b.errors > 0)
$display("PASS: the checker passes the correct FIFO and reports the buggy one");
else $display("FAIL: checker verdicts are wrong");
$finish;
end
initial begin #100000; $display("FAIL: watchdog timeout"); $finish; end
endmoduleIt prints:
correct FIFO: 160 pops checked, 0 errors, 0 leftovers
buggy FIFO: 163 pops checked, 45 errors, 0 leftovers
PASS: the checker passes the correct FIFO and reports the buggy oneHow to read it: the checker compared 160 pops on the correct FIFO and found nothing wrong, with no words left over; on the buggy FIFO it found 45 wrong words, every one caused by a fifth push overwriting the oldest entry. A print-only testbench would have produced two equally plausible logs. The final line is a single verdict a regression script can read, and it holds only because the checker demonstrably fails on a real bug.
Count errors, report the first few in detail: enough to debug, not a flood.
Fail loudly on anything unexpected: an output with no prediction, a prediction never matched, a timeout.
Print exactly one verdict line: scripts match it; anything else is a failure.
Keep the reference independent of the design: same specification, different implementation (lesson 3.4).
Prove the checks can fail: plant a bug, as here and in the Bug Zoo, or use mutation testing (lesson 6.8).
Practice | Purpose |
|---|---|
Scoreboards and monitors in every environment | Every output is compared automatically (lesson 5.4) |
Error counters and a summary report at the end | One verdict per test, readable by regression tools |
Global timeouts in every test | Hangs become failures, not stuck jobs |
Bug seeding or mutation campaigns | Evidence that the checkers detect real bugs |
A team (illustrative) inherited a regression where each test printed "TEST DONE" at the end and the script treated any run that printed it as a pass. Several tests' checkers had been commented out during debug months earlier. The tests kept printing "TEST DONE". The checker-free tests were found only when a bug they should have caught reached a later stage.
To show the checker detects a real bug; a checker that passes the correct FIFO might also pass everything.
A test whose checks never ran or were disabled still prints it and passes; the verdict must come from the checks, with errors counted.
A test whose verdict needs a person reading the output is a demonstration.
Self-checking means a reference, a comparison on every output, end-of-test checks, a watchdog and one verdict.
Show the checks can fail, with a planted bug or mutation.