Every verification metric answers a narrower question than the one people ask of it. Coverage answers "did these bins get hit", not "is the design correct". A pass rate answers "did the checks fire", not "are the checks any good". A principal engineer knows what each number cannot tell you.

Coverage that mirrors the wrong thing

A coverage model is only as good as its source. If the bins are copied from the RTL structure or from the stimulus generator, they measure what the bench does, not what the spec requires. They reach 100% easily, and a bug can sit outside every bin.

The bench below is an illustrative case. An 8-bit ALU has eight operations. The original test kept operands below 128 "to avoid overflow noise", and its coverage crosses each operation with the operand ranges the generator produces. That is a mirror of the generator.

A second, feature-based model asks what the spec cares about: an add that carries out, a subtract that borrows, a left shift that loses a 1, a right shift of a value with bit 7 set. Verilator 5.048 ignores covergroups (it warns COVERIGN), so both models are hand-written bin arrays; the covergroup equivalent is shown after the bench.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

package alu_pkg;
  typedef enum logic [2:0] {ADD, SUB, AND_, OR_, XOR_, SHL, SHR, PASSB} op_t;
endpackage

module alu8 import alu_pkg::*; (
  input  op_t        op,
  input  logic [7:0] a,
  input  logic [7:0] b,
  output logic [7:0] y,
  output logic       c      // carry, borrow, or the bit shifted out
);
  always_comb begin
    c = 1'b0;
    case (op)
      ADD:   {c, y} = {1'b0, a} + {1'b0, b};
      SUB:   {c, y} = {1'b0, a} - {1'b0, b};
      AND_:  y = a & b;
      OR_:   y = a | b;
      XOR_:  y = a ^ b;
      SHL:   {c, y} = {a, 1'b0};
      SHR:   {y, c} = {1'b0, a};        // logical: zero enters at the top
      default: y = b;
    endcase
  end
endmodule

With the correct ALU, phase 1 ends with mirror coverage at 100% and feature coverage at 75%. Phase 2 closes the feature bins and the bench passes. Now plant a bug: make the right shift arithmetic, so bit 7 is copied instead of cleared.

// snippet: the planted bug
SHR: {y, c} = {a[7], a};   // should be {1'b0, a}

The run with the bug (Verilator 5.048) prints phase 1: mirror coverage 100%, feature coverage 75%, wrong results 0, then 22 wrong results in phase 2, all on SHR with bit 7 set, and ends in FAIL. Phase 1 is the original regression: green, at 100% coverage, with a live bug. A second mutant, deleting phase 2, also fails: the "feature holes remain" check stops the bench from signing off at 75%.

In a real environment the feature model is a covergroup. A sketch of the same bins:

1
2
3
4
5
6
7
8
9
10

// snippet: covergroup form of the feature model
covergroup alu_features with function sample(op_t op, logic [7:0] a, logic [7:0] b);
  cp_op : coverpoint op;
  cp_corner : coverpoint {op, a[7]} {
    bins shl_msb = {{SHL, 1'b1}};
    bins shr_msb = {{SHR, 1'b1}};
  }
  cp_carry  : coverpoint (op == ADD && (9'(a) + 9'(b)) > 255) { bins hit = {1}; }
  cp_borrow : coverpoint (op == SUB && a < b)                 { bins hit = {1}; }
endgroup

Signs that coverage is a mirror

• Bins sample RTL internals such as decoded state, instead of transactions seen at the interface.

• Bin boundaries match constraint boundaries in the sequence code.

• Coverage reached 100% in the first week and never had a hole worth discussing.

• Nobody can point to the spec section behind a bin.

• The same person wrote the RTL decode, the stimulus constraints and the coverage, from one reading of the spec.

The cure is to write the coverage model from the spec and the verification plan, before looking at the RTL or the sequences. See From Spec to Coverage Plan.

Flat bug curves

A bug-discovery curve that goes flat is read as "the design is stable". It can also mean the bench has stopped exploring. Before reading a flat curve as maturity, check these:

A useful test is to plant a few bugs (mutation testing) and see whether the regression catches them. If it misses planted bugs, a flat curve says nothing.

Pass-rate traps

• 98% passing can hide one real bug behind 2% of failures that everybody calls "infrastructure". Group failures by first-error signature and look at each signature, not at the percentage.

• 100% passing after a change that disabled a checker is worse than 95% with the checker on. Track the number of checks performed, not only the number that failed.

• A test that cannot fail is common: an empty scoreboard queue at the end, a monitor that never saw a transaction, an assertion with a vacuous antecedent. Every test should report how many items it compared, and fail on zero.

• Waived failures pile up. Each waiver needs an owner and an expiry, or the pass rate becomes a measure of how many waivers exist.

Related topics

• Previous: Intermittent Failures

• Next: Scaling Verification

• Sign-Off Metrics and Review

• When Passing Is the Bug: Vacuity & Coverage