In a mature project the RTL has survived thousands of runs and the testbench changed today. The new failure is statistically a testbench fault.
The worst testbench faults never fail. They stop checking, and the regression goes green while the DUT goes unverified.
All-pass-too-easily: a stubborn test passes first try after an unrelated change.
Scoreboard processed zero transactions while the log says pass.
Assertions pass in thousands with an antecedent that never fired.
Coverage drops or flatlines across the regression.
Identical counts across wildly different seeds.
The end-of-test gate below is pseudocode for any methodology: it refuses a quiet pass. Gate on activity, not just absence of complaints.
END-OF-TEST SANITY GATE (pseudocode) — refuse a silent pass
scoreboard.compare_count > 0 else FATAL "SB compared nothing"
monitor.txn_count > 0 else FATAL "monitor saw nothing"
cover-hits(antecedent) > 0 else WARN "assertion never armed"
expected_q.size() == 0 else ERROR "N expected never matched"
A pass with zero activity is a fail wearing a green shirt.Permanent counters make hangs mappable and silent death visible. Print them at end of test and on a heartbeat during the run.
module cnt_dut (
input logic clk,
input logic [7:0] d,
output logic [7:0] q
);
always_ff @(posedge clk) q <= d + 8'h01;
endmodule
The bench drives eight directed vectors with no solver. The heartbeat maps thread progress, and the gate turns a zero-compare pass into a loud failure.
Layer 0, always on: per-component counters printed at end of test.
Layer 1, verbosity switch: one line per transaction at each handoff.
Layer 2, debug builds only: full dumps inside the failure window.
Discipline: instrumentation is permanent infrastructure, not throwaway prints.
Make the other silent killers loud as well. Validate handles where they enter a component, and always check the randomize return, since a failed call keeps stale values. Solver depth belongs to the randomization material, mailbox rules to the interprocess material.
Fail. A pass with zero activity is a checker that never armed.
Stale values from the previous object state. The stimulus looks live but repeats.
One seed in a thousand fired an illegal-burst assertion. A deterministic random failure is the best kind: it reproduces exactly.
TIMELINE — Case 2 (uninitialized config knob)
T+0:00 Seed 4217 fails: "burst must not cross 4KB" fires.
Assertion read first — it encodes the spec correctly.
Stimulus is genuinely illegal → TB generated it.
T+0:15 Where do burst bounds come from? cfg.max_burst_bytes,
set by the test from a +arg... supposedly.
T+0:30 Log the cfg object at start: max_burst_bytes = 0, never set.
The +arg parse populated a LOCAL variable; the copy into
cfg happened before the parse on most orderings — a build
refactor had moved the parse later for one test variant.
T+0:45 Why only seed 4217? With max=0 the bound constraint
became insoluble; randomize() returned 0 — UNCHECKED —
leaving len/size at stale values from the previous txn.
Most stale pairs were legal by luck; seed 4217 produced
a stale pair crossing 4KB.
T+1:00 Fixes (all three): check every randomize() return;
fatal on cfg fields still at poison defaults
(init to a poison constant, not 0); end-of-build print.Generalized lesson: a seed failure is a latent deterministic fault plus a stimulus pattern. Poison defaults turn this class into a first-run fatal.
A stable test timed out with the DUT idle. Hang signature means thread-dump approach: map every blocked thread before theorizing.
TIMELINE — Case 3 (objection and drain logic)
T+0:00 Watchdog timeout at 2ms; DUT idle since 0.4ms. No complaints.
T+0:10 Heartbeat counters: drv_sent=200/200, mon_seen=200,
sb_cmp=198. Two responses outstanding... forever.
T+0:30 Thread dump: scoreboard blocked in expected-queue drain
wait (waiting for 2 more matches); test's end-of-test
objection held by the scoreboard until the queue empties.
Classic standoff: nothing will produce 2 more responses.
T+0:50 Why 2 missing? Monitor counts 200, scoreboard compared
198 → drop INSIDE the scoreboard path. A hand-rolled
subscriber did try_put into a bounded uvm_tlm_fifo (64);
under a late burst 2 try_puts failed, return unchecked.
T+1:10 Fix: use uvm_tlm_analysis_fifo (unbounded), or check the
try_put return; never drop on a checking path. Drain logic gains a
progress watchdog: if the outstanding count is static for
100us, stop with the queue contents printed.Generalized lesson: end-of-test hangs are accounting faults. Drain logic must report what it awaits, and checking paths must never drop silently.