Your test passes. How do you know it would fail if the design were wrong? That question sounds strange the first time you hear it. A green test feels like good news, and the screen says what you hoped to see.
But a test that passes on broken code is worse than no test at all. It tells you to stop looking. This lesson shows how to check the test itself, using small broken copies of your own design.
A green test says one thing: the design survived the cases the test contains. It says nothing about the cases the test left out. Every testbench has blind spots, and the design bugs hiding in those blind spots are the ones that reach silicon.
The way out is to attack the testbench on purpose. Take a copy of the design, change one thing in it, and run the test. If the test fails, it killed the mutant. If the test still passes, the mutant survived, and you have learned something about your test.
A mutant is a copy of the design with one deliberate change. A good test fails on every mutant that is a real bug. An equivalent mutant is the exception: it behaves identically to the design on every input, so a test is right to pass it. Surviving mutants are guilty until proven equivalent.
The design is the course running example, a 4-bit event counter. Each rising clock edge does the first of these that applies: reset (active low, synchronous) forces the count to 0, then clear forces it to 0, then a valid event adds 1 with wrap from 15 to 0, then the count holds. The full contract lives in the spec at docs/spec.md, and both testbenches check against that spec, never against the design code.
Two testbenches face nine variants of this counter. The 8-edge directed test is the testbench from lesson 4.2: eight fixed stimulus edges with hand-written expected values. The strong test is the lesson 5.2 testbench: it checks every input combination each cycle against an independent model of the spec.
The table below lists all nine variants and what each testbench did with them, measured on macOS, Icarus Verilog 13.0. Read the middle column first: seven variants are real bugs and two are not.
Variant | What it changes | Real bug? | 8-edge directed test from lesson 4.2 | Strong test (lesson 5.2) |
|---|---|---|---|---|
| none | no | pass | pass |
| clear checked before reset | no, equivalent (both give 0) | pass | pass |
| reset checked after count | yes | MISSED | caught |
| count checked before clear | yes | caught | caught |
| count falls to 0 when idle instead of holding | yes | caught | caught |
| stops at 15 instead of wrapping | yes | MISSED | caught |
| no reset branch | yes | caught | caught |
| adds 2 | yes | caught | caught |
| reset acts without waiting for the clock edge | yes | MISSED | caught |
The directed test catches 4 of the 7 real bugs and misses 3; the strong test catches all 7 and correctly passes the equivalent one.
The figure below draws the same result as two rows of seven cells, one cell per real bug. A marked cell means the testbench caught that bug; the first row names the three it missed.
`timescale 1ns/1ps
// Mutant reset_last: reset checked last, so clear and valid can beat reset.
module mini_controller (input logic clk, rst_n, event_valid, clear_count, output logic [3:0] count);
always_ff @(posedge clk) begin
if (clear_count)
count <= 4'd0;
else if (event_valid)
count <= count + 4'd1;
else if (!rst_n)
count <= 4'd0;
end
endmoduleThis mutant checks reset last, so a cycle with reset asserted and an event present counts instead of returning to 0.
`timescale 1ns/1ps
// Mutant saturates: count sticks at 15 instead of wrapping to 0.
module mini_controller (input logic clk, rst_n, event_valid, clear_count, output logic [3:0] count);
always_ff @(posedge clk) begin
if (!rst_n)
count <= 4'd0;
else if (clear_count)
count <= 4'd0;
else if (event_valid)
count <= (count == 4'd15) ? count : count + 4'd1;
end
endmoduleThis mutant holds the count at 15 instead of wrapping to 0 on the next event.
The directed test catches four mutants. Each message below names the edge that exposed the bug and the expected value from the spec. Take them one at a time and match each to the stimulus edge that produced it.
# captured terminal output
ERROR edge 4: expected count=0 got 4Edge 4 drives clear high with an event present and expects clear to win with 0. The valid_before_clear mutant counts first, so the count moves from 3 to 4 and the check reports it.
# captured terminal output
ERROR edge 6: expected count=1 got 0Edge 6 is the first idle edge: no event, no clear, so the count should hold at 1. The idle_clears mutant drops it to 0 instead, and the hold check reports it.
# captured terminal output
ERROR edge 0: expected count=0 got xEdge 0 holds reset and expects 0. The reset_ignored mutant has no reset branch, so the count register never takes a known value and the check reads x, the unknown value.
# captured terminal output
ERROR edge 1: expected count=1 got 2Edge 1 is the first counting edge and expects 1. The counts_by_two mutant adds 2, so the very first increment reports it.
The strong test catches all seven, including the three the directed test misses. Its messages name the full input combination and the simulation time, so each one points at the exact spec row the mutant breaks.
# captured terminal output
ERROR: rst_n=0 clear=0 valid=1: expected 0 got 4 at 126000This stimulus asserts reset with an event present and expects reset to win with 0. The reset_last mutant counts first and reports 4, which exposes the broken priority.
# captured terminal output
ERROR: rst_n=1 clear=0 valid=1: expected 0 got 15 at 376000This stimulus counts with the counter at 15 and expects the wrap to 0. The saturates mutant sticks at 15, which exposes the missing wrap.
# captured terminal output
ERROR: count changed before the clock edge (x -> 0)This check watches the count between clock edges. The async_reset mutant drives the count to 0 the moment reset falls, before any edge arrives, which exposes reset acting without the clock.
All three. Reset is exercised exactly once, on edge 0 with no event present, so broken reset priority never meets exposing stimulus. The count climbs only to 3, so it never reaches 15 and the missing wrap never shows. Reset is only checked after a clock edge, so acting between edges looks identical at every check.
Now cause a failure yourself. The mutants folder holds complete replacements for the design file. Pick reset_last, the mutant the directed test cannot see, so the strong test is the one that must speak up.
Copy mutants/reset_last.sv over rtl/mini_controller.sv in the example project, keeping a copy of the original elsewhere.
Run make sim and watch it fail: the strong test reports the reset priority error and make exits with a nonzero status.
Restore the original design file and run make sim again to confirm the run is green before you continue.
The 8-edge test misses reset_last because it never asserts reset while an event is present. You can close that gap with one more edge. The file is counter_wave.sv from lesson 4.2, in the lesson-files/ folder of start-here-examples.zip.
Open counter_wave.sv and find the line if (errors == 0) near the end of the second initial block.
Just above it, add one edge: drive reset low with an event high, wait for the rising edge and 1 ns, and expect 0. The two lines are in the box below.
Run it on the good design and confirm it still passes.
In the same file, replace the body of the design's always_ff block with the body from the reset_last mutant shown above, and run again. The test must now fail.
// snippet: the new edge, added above if (errors == 0)
@(negedge clk); rst_n = 0; event_valid = 1; clear_count = 0;
@(posedge clk); #1; check(4'd0, 8); // edge 8: reset beats an event -> 0# captured terminal output
$ iverilog -g2012 -s tb -o good.vvp counter_wave.sv && vvp good.vvp
VCD info: dumpfile wave.vcd opened for output.
PASS: first simulation
$ iverilog -g2012 -s tb -o bad.vvp counter_wave.sv && vvp bad.vvp
VCD info: dumpfile wave.vcd opened for output.
ERROR edge 8: expected count=0 got 2
FATAL: counter_wave.sv:66: FAIL: 1 mismatches
Time: 86000 Scope: tbThe first run passes on the good design. The second fails at edge 8 with expected count=0 got 2: the mutant let the event win over reset. One added edge turned a surviving mutant into a killed one. That is the whole method: find a surviving mutant, name the stimulus it needs, add it, and see it die. saturates needs the count to reach 15, which takes about fourteen more counting edges, so a loop is the practical way to write that test.
Counting passes instead of kills: a testbench that passes 100 runs but never faced a mutant has proven nothing. Score your testbench by the mutants it kills, not by its green runs.
Writing tests by reading the design instead of the spec: a test copied from the design repeats the design's mistakes and passes on both. Write expected values from docs/spec.md, as both testbenches here do.
Treating every surviving mutant as a test hole: swap_reset_clear survives both testbenches and that is the correct verdict, because both branches assign 0. Prove a survivor equivalent before you close it.
Mutating the test instead of the design: weakening a check until the mutant passes hides the bug in both. Change only the design copy, never the expected values.
Rule: a test counts as a test only after you have watched it fail on a real bug.
Assert reset while event_valid is high: drive rst_n low with event_valid high and clear low, wait for the rising edge, then expect 0. The original design gives 0 because reset has priority; the reset_last mutant counts instead, so the new check fails on the mutant and passes on the design.
No. It is an equivalent mutant: reset and clear both assign 0, so no input sequence can distinguish it from the original. A test is right to pass it, and the strong test correctly does.
That this test can fail: it kills that one mutant. Nothing more. Run the full mutant set before claiming the testbench is strong, because saturates and async_reset each need their own stimulus.
A green test only proves the design survived the cases it contains; mutation testing checks the test itself.
A good test fails on every mutant that is a real bug and passes an equivalent mutant.
Name every surviving non-equivalent mutant, add the stimulus it needs, and keep the mutant run green.