A refactor lands on Friday. On Monday every test is green, and the regression runs a little faster than before. Nobody complains about a green regression, so nobody looks. Three weeks later a silicon-bound bug is traced to a block whose tests have passed every night since that Friday, without comparing a single word. A false pass is the most dangerous failure in verification because it does not look like a failure. How do you make a pass prove that something was checked?
A smoke detector with no battery is perfectly quiet. Silence from a checker means "nothing wrong" only if the checker was running, was fed, and was able to object. So a trustworthy PASS needs three kinds of evidence beyond UVM_ERROR : 0: the checkers saw traffic (a count above zero), nothing that should have been an error was demoted without a reviewed reason, and the checkers can fail (shown by planting a bug and watching them catch it).
dma_early_end_test starts its packet in the background and drops its objection at once, so the run ends before any data moves. The scoreboard's check_phase guard catches it: zero words compared is an error. dma_false_pass_test is the same test after someone "quietened" the scoreboard's end-of-test errors with two unconditional severity overrides:
// snippet: from the course DMA bench
// False-pass playbook: the same early end, after someone "quietened" the
// scoreboard's end-of-test errors for the whole env. The verdict says PASS
// although the test compared nothing.
class dma_false_pass_test extends dma_early_end_test;
`uvm_component_utils(dma_false_pass_test)
function new(string name, uvm_component parent);
super.new(name, parent);
endfunction
function void end_of_elaboration_phase(uvm_phase phase);
super.end_of_elaboration_phase(phase);
env.sb.set_report_severity_id_override(UVM_ERROR, "SB_EMPTY", UVM_INFO); // BUG: unconditional,
env.sb.set_report_severity_id_override(UVM_ERROR, "SB_LEFTOVER", UVM_INFO); // unreviewed waivers
endfunction
endclass+UVM_TESTNAME=dma_early_end_test
UVM_ERROR dma_tb.sv(407) @ 155000: uvm_test_top.env.sb [SB_EMPTY] no words were compared: the test checked nothing
UVM_INFO dma_tb.sv(411) @ 155000: uvm_test_top.env.sb [SB] 0 words matched, 0 mismatched
UVM_ERROR : 1
UVM_FATAL : 0
TEST VERDICT: FAIL (UVM_ERROR=1, UVM_FATAL=0)
+UVM_TESTNAME=dma_false_pass_test
UVM_INFO dma_tb.sv(407) @ 155000: uvm_test_top.env.sb [SB_EMPTY] no words were compared: the test checked nothing
UVM_INFO dma_tb.sv(411) @ 155000: uvm_test_top.env.sb [SB] 0 words matched, 0 mismatched
UVM_ERROR : 0
UVM_FATAL : 0
TEST VERDICT: PASS (UVM_ERROR=0, UVM_FATAL=0)The two logs are identical except for one word. The SB_EMPTY message is still printed in the false pass, now as UVM_INFO, and the scoreboard still says 0 words matched. Every piece of evidence was in the log; the verdict ignored it, because the verdict counts only errors and fatals. That is the general shape of a false pass: the evidence exists, but nothing turns it into a failure.
False pass | On this bench | What the log shows | Defence |
|---|---|---|---|
Nothing checked |
|
| A check_phase guard that fails on zero compares, never demoted globally |
Errors demoted or waived broadly |
| The error text appears as UVM_INFO | Waivers scoped to one ID, one pattern, one component, with a ticket and a count |
A comparator that matches everything |
|
| Mutation runs: plant a bug and require every checker to fail |
Coverage counting the wrong thing | War story 8: a W1C clear counted when the bit was never set | Coverage closes; the scenario never happened | Sample only the condition the plan item means; review bins against the plan |
Checks after a fatal | A PASS banner printed from report_phase | No banner, only the library's report summary; a script that greps for PASS reads the missing banner as silence, and silence as success | Print the verdict from |
A test that ended early |
| End time far shorter than the scenario | Owed-work counts and end-of-test checks (next lesson) |
dma_cmp_trap_test feeds a deliberately wrong predictor into uvm_in_order_class_comparator, which compares with compare(). axis_item has no do_compare(), so every compare succeeds. The trap is worse than it sounds. Built with the planted word-6 bug, the same test shows the scoreboard catching the corrupted word while the class comparator logs the very same corrupted value as a match:
+UVM_TESTNAME=dma_cmp_trap_test with EXTRA_ARGS=+define+DMA_BUG_WORD6 (scratch run)
UVM_INFO @ 335000: uvm_test_top.env.class_cmp [Comparator Match] data=3d908879
UVM_ERROR dma_tb.sv(396) @ 375000: uvm_test_top.env.sb [SB_MISMATCH] expected 52c7d685, got ad38297a
UVM_INFO @ 375000: uvm_test_top.env.class_cmp [Comparator Match] data=ad38297a
UVM_INFO @ 395000: uvm_test_top.env.class_cmp [Comparator Match] data=88390659
UVM_INFO dma_tb.sv(411) @ 655000: uvm_test_top.env.sb [SB] 7 words matched, 1 mismatched
UVM_INFO dma_tb.sv(1637) @ 655000: uvm_test_top.env [CMP] class comparator: 8 matches, 0 mismatchesA comparator that cannot fail is not a weak check; it is a false statement printed eight times. The only way to find one is to give it something it must reject.
dma_waiver_test is the bench's scoped waiver: a report catcher that demotes only SB_MISMATCH, only on env.sb, tags each message with ticket DMA-142 and counts them. It exists because the model has a known limitation (war story 9: the DMA ignores KEY writes while busy, the scoreboard does not), and its five waived mismatches are expected. Now build it with the planted bug and compare one line:
+UVM_TESTNAME=dma_waiver_test (normal build)
UVM_INFO dma_tb.sv(396) @ 435000: uvm_test_top.env.sb [SB_MISMATCH] [WAIVED DMA-142] expected 089dd98a, got f762d975
+UVM_TESTNAME=dma_waiver_test with EXTRA_ARGS=+define+DMA_BUG_WORD6 (scratch run)
UVM_INFO dma_tb.sv(396) @ 435000: uvm_test_top.env.sb [SB_MISMATCH] [WAIVED DMA-142] expected 089dd98a, got 089d268a
UVM_INFO dma_tb.sv(411) @ 655000: uvm_test_top.env.sb [SB] 3 words matched, 5 mismatched
TEST VERDICT: PASS (UVM_ERROR=0, UVM_FATAL=0)Word 6's actual value changed (f762d975 became 089d268a) because the DUT bug corrupted it, but that word was already one of the five expected mismatches, so the waiver swallowed it and the verdict stayed PASS. Even the waiver's count did not move. A waiver over a whole message ID blinds that checker to every bug that produces the same message. Scoping helps, but the real fix is to remove the cause, here by correcting the model, and to give every waiver an owner and an expiry date.
Evidence-of-checking counters. Every checker counts what it compared and fails if the count is zero or below what the test asked for. The bench's SB_EMPTY guard is the minimum; a stricter one compares the count with the words the DUT was asked to move.
Scoped, ticketed waivers. Demote by ID and message pattern on one component, tag the message with the ticket, count it, and fail if the count changes. Never set_report_severity_id_override across an env without review; there is no _hier version of it in UVM 1.2 or IEEE 1800.2, and that is a feature.
Mutation runs. Compile the DUT with a planted bug (here DMA_BUG_WORD6) and run the regression. Every test that can reach the bug must fail. A test that passes is either out of reach of that bug (fine, and worth knowing) or blind (a defect in the bench).
A verdict line that survives fatals. Print it from report_summarize, and have the regression script treat a log without a verdict line as a failure.
This is what the planted bug did to the bench's own regression (scratch runs, one build with the define):
Test | Result with DMA_BUG_WORD6 | What it means |
|---|---|---|
| FAIL: 7 matched, 1 mismatched | Caught |
| FAIL: 13 matched, 1 mismatched (the 10-word job) | Caught; the 1- and 3-word jobs cannot reach word 6 |
| FAIL: scoreboard and in-order comparator each 13 and 1 | Two independent checkers caught it |
| Scoreboard FAIL; class comparator 8 matches, 0 mismatches | The class comparator is blind |
| PASS: 4 matched | Out of reach: only 4 words reach the DMA |
| PASS: no data moves | Out of reach by design |
| PASS: the bug hides under DMA-142 | Blind: the waiver covers the bug |
| PASS: 0 words compared | Blind: checks nothing |
Two tests out of reach, two blind spots and one blind checker: an afternoon's mutation run found three bench defects that months of green regressions had hidden.
env.sb.set_report_severity_id_override(UVM_ERROR, "SB_LEFTOVER", UVM_INFO) to a base test because one test leaves words in flight at the end. What is wrong with it, and what would you do instead?It disables the leftover check for every test derived from that base, so any test that loses data at the end now passes. Fix the one test's end of test (wait for the DONE interrupt and the drain), or, if the leftover is genuinely expected, demote it in that test only, with a pattern for the expected count, a ticket and a counter that fails if it changes.
Either the stimulus never reaches the bug (for example, the bug needs a 6-word job and the test only sends 4-word jobs), or the checker is blind to it (a waiver or a comparator that ignores the field). Check whether the bug's trigger condition occurred in that run (coverage of the trigger, or a message from the planted code); if it did and the test still passed, the checker is blind.
UVM_ERROR lines. Which false pass does that let through that a verdict line from report_summarize would catch?A run that died on a fatal before its summary (or crashed, or was killed by the farm): no UVM_ERROR lines were printed, so the script calls it a pass. Requiring a TEST VERDICT: PASS line, printed from report_summarize, turns a missing verdict into a failure.
Next: one of the false passes deserves a playbook of its own. Early-End Playbook shows why a test can finish in 155 ns and how to tell that from a fast pass.