A failing test is good news: something is wrong and the testbench noticed. A passing test is ambiguous: either the design is right, or the testbench was not looking. Most of the time nobody asks which. Every DV engineer eventually meets a regression that was green for months while one of its checkers checked nothing at all.

How do you know your checker actually works?

The obvious answer, and why it breaks

The obvious evidence is that the checker exists and the tests pass. But a check passes in three situations: the design is right, the check's condition never occurred, or the check was never able to fail. Only the first is evidence. An assertion req |-> ack passes in every cycle where req is low, so a test that never raises req passes it perfectly.

The idea: a check must be triggered, and must be able to fail

Build it: a vacuous pass, exposed and fixed

The rule: every req is followed by ack within 1 to 3 cycles. Phase 1's stimulus has a bug, a wrong signal name, so req never rises. Phase 2 fixes it. Phase 3 breaks the design on purpose so that ack never comes. A counter records how often the assertion's trigger occurred in each phase.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

// Lesson 5.8: a false pass, and how to catch one. The rule: every req is followed by ack within
// 1 to 3 cycles. Phase 1's stimulus has a bug: it drives `start`, which nothing reads, instead
// of req, so the assertion is never triggered and passes vacuously. Phase 2 fixes the stimulus
// and counts how often the assertion's trigger occurred. Phase 3 breaks the design on purpose
// (ack never comes) to prove the assertion can fail.
module tb;
  logic clk = 0;
  always #5 clk = ~clk;
  logic req = 0, start = 0, ack = 0, r1 = 0, break_ack = 0;
  always @(posedge clk) begin r1 <= req; ack <= r1 && !break_ack; end   // design: ack 2 cycles after req

  int fails [1:3] = '{0, 0, 0}, triggers [1:3] = '{0, 0, 0};
  int phase = 1;
  a_ack: assert property (@(posedge clk) req |-> ##[1:3] ack) else fails[phase]++;
  always @(posedge clk) if (req) triggers[phase]++;  // how often the assertion was really tested

  task automatic pulses(input bit use_req);
    repeat (20) begin
      @(posedge clk); if (use_req) req <= 1; else start <= 1;
      @(posedge clk); req <= 0; start <= 0;
      repeat (4) @(posedge clk);
    end
  endtask

  initial begin
    phase = 1; pulses(0);                            // the buggy test: drives start, not req
    phase = 2; pulses(1);                            // fixed
    phase = 3; break_ack = 1; pulses(1);             // design broken on purpose
    repeat (5) @(posedge clk);
    for (int p = 1; p <= 3; p++) $display("phase %0d: assertion triggered %0d times, failed %0d times", p, triggers[p], fails[p]);
    if (fails[1] == 0 && triggers[1] == 0 && fails[2] == 0 && triggers[2] == 20 && fails[3] > 0)
      $display("PASS: phase 1 passed by checking nothing; phase 2 checked 20 requests; phase 3 proved the assertion can fail");
    else $display("FAIL: unexpected counts");
    $finish;
  end
endmodule

It prints:

phase 1: assertion triggered 0 times, failed 0 times
phase 2: assertion triggered 20 times, failed 0 times
phase 3: assertion triggered 20 times, failed 20 times
PASS: phase 1 passed by checking nothing; phase 2 checked 20 requests; phase 3 proved the assertion can fail

How to read it: phase 1 passed with zero failures and zero triggers: the assertion was never tested. Without the trigger counter, phases 1 and 2 would look identical in the log. Phase 3 shows the other half of the evidence: when the design is wrong, the assertion fails, 20 times out of 20. A checker earns trust only with both numbers, triggered and able to fail.

Proving checkers work, at scale

• Count triggers and comparisons, and fail a test whose count is zero or suspiciously low.

• Cover the antecedent of every assertion (Part 6), so vacuity shows in the coverage report.

• Plant bugs: the Bug Zoo in this course, mutation testing in general (lesson 6.8).

• Validate checkers on the clean design across many seeds: lesson 5.5's arbitration checker passed its first seed and failed the clean design in 17 of 30.

• Review what every check depends on: a check built from the signal under suspicion cannot catch its bug (lesson 3.4).

How real teams guard against false passes

What breaks

In lesson 5.3, DCRouter's R1 assertion (a disabled port is never ready) passed every run. The random hunt never disables a port, so the condition the assertion guards never arose: it was true every cycle without ever being tested. (Strictly it is not vacuous, since it has no antecedent; the same false pass happens, and covering "a port is disabled" is the remedy.) BUG_23, which breaks exactly R1, escaped every test so far. The assertion was correct; the evidence that it worked was missing. Part 6's coverage model makes that gap visible.

Key takeaways

• A pass means the design is right only if the check was triggered and could have failed.

• Count triggers and comparisons, cover assertion antecedents, and plant faults to prove checkers fail.

• Validate checkers on the clean design across many seeds before trusting them.

Related topics

• Part 5 overview: Checking