The sign-off review for the DMA is in an hour. Your slide says "pass rate 100%, functional coverage 100%, code coverage 98%". The design lead asks three questions: how many tests had their waivers reviewed, what are the 2% of code nobody executed, and is there an open bug whose fix is not yet in the netlist? The numbers on the slide answer none of them. What does a sign-off review actually need, and how do you present it?
No single metric says a block is verified. A pass rate says the bench found nothing wrong tonight, not that it looked everywhere. Functional coverage says the scenarios in the plan happened, not that they were checked. Code coverage says the RTL was exercised, not that the results were compared. Bug rates say whether the bench is still finding bugs. Each number covers a hole in the others, and a reviewer reads them together, over time, along with the things that are not numbers: reviewed waivers, open bugs and the status of other verification (assertions, formal, gate-level and power-aware simulation).
A doctor's check-up works the same way. Blood pressure, a blood test and an ECG each look at a different system; a patient can pass one and fail another. And a single reading means less than the trend: a value that has been stable for a year means something different from one that just jumped.
Metric | What it shows | How it misleads |
|---|---|---|
Pass rate (per night, trended) | Whether the design and bench are stable | A bench that checks nothing passes 100%; waived and demoted errors do not count. See the false-pass playbook. |
Functional coverage (merged) | Plan scenarios that occurred | Only as good as the plan; a sampled bin does not prove the result was checked |
Code coverage (merged) | RTL lines, branches, expressions and toggles exercised | Exercised is not checked; 100% says nothing about missing features |
Bug find rate | Whether the bench is still finding bugs | Drops when the bench stops looking, not only when the design is clean |
Bug close rate and open bugs by severity | Whether fixes keep up | A "fixed" bug is fixed only when the fix is in the build being signed off |
Stability (flaky and crash rate) | Whether results can be trusted | Rerun-on-fail hides it |
Waivers and exclusions | What was deliberately left out | An unreviewed waiver is a hidden failure |
Trends matter more than snapshots. A healthy project shows the bug find rate rising early (the bench is new and finds a lot), peaking, then falling to near zero for several weeks while regression volume stays the same. Coverage climbs fast, then flattens into a long tail that only directed tests close. A find rate that falls to zero the same week the nightly was cut in half is not convergence; it is the bench looking less.
Here is the DMA block's sheet as it stands with the course bench, filled in from runs on this and earlier pages. It is not a passing sheet, and that is the point: a real sheet shows the gaps, each with a reason and an action.
Item | Status | Evidence | Action |
|---|---|---|---|
Regression pass rate | 61 of 61 nightly runs pass on the clean build | regression infrastructure manifest | Trend it nightly |
Bench sensitivity | The planted word-6 bug fails 26 of the 30 regular runs in the triage nightly; the 4 that pass cannot reach word 6 | Keep a mutation build in the weekly | |
Functional coverage | 17 of 17 plan items in the merged union; | Keep the closure tests in the nightly | |
Code coverage (Verilator) | Lines 15 of 17, branches 40 of 43, expressions 16 of 18, toggles 384 of 452 | Unmapped-address test; write-while-busy test | |
Waivers | 1: DMA-142, 5 SB_MISMATCH messages in | War story 9: the model ignores the busy-write rule | Fix the predictor, then remove the waiver |
Open bugs | None in the DUT; one bench limitation (DMA-142) | Triage history | Close DMA-142 before sign-off |
Assertions | None in the block bench | Add APB and AXI-Stream protocol assertions | |
Stability | Plain build 0 of 20 crashes; coverage build 8 of 20 | Ticket against the coverage build; use only complete runs | |
GLS and power | Not run in this course | Out of scope here; on a real project, required |
A reviewer reading that sheet signs off nothing yet, and knows exactly why: one open bench limitation with a waiver, two code-coverage holes that point at untested features, no assertions, and an unstable coverage build. That is a useful review. A slide saying "100% pass, 100% functional coverage" would have been true for this same bench, and would have hidden every one of those items.
Plan coverage: every plan item maps to coverage or a test, and every item is hit in the merged result of passing runs on the sign-off build.
Code coverage holes explained: every uncovered line, branch, expression and toggle is either covered, excluded with a reviewed reason, or turned into a plan item and a test.
Waivers reviewed: each waiver names its ticket, its exact scope (component, ID, pattern) and its count, and a reviewer other than its author has approved it.
No open critical or major bugs, and every closed bug's fix is in the build being signed off, with a guard test that would fail if it came back.
Bug rate converged: no new DUT bugs for an agreed period at full regression volume.
Regression stable: flaky and crash rates are known and ticketed, and the sign-off numbers come only from complete runs.
Assertions: protocol and design assertions are enabled in regression, with their own pass and cover counts.
Other verification: formal results, gate-level simulation and power-aware simulation status, where the project requires them.
Lead with the decision and the gaps, not with the green numbers. "Not ready: three items, all with owners and dates" is a better first line than a coverage chart.
Show trends, not snapshots: pass rate, coverage and bug find rate over the last several weeks, on the same axis of time, with regression volume shown alongside.
Give every number its denominator and its source: "15 of 17 lines, merged from six tests, Verilator line coverage" can be checked; "88%" cannot.
List every waiver and exclusion with its reason and reviewer. They are where reviewers find problems.
A project's weekly report showed the DUT bug find rate falling from twelve a week to zero over a month, and the team proposed sign-off. A reviewer asked for the regression volume on the same chart: it had fallen by two thirds over the same month, because a farm migration had dropped most of the nightly's random seeds and nobody had noticed. Restoring the full nightly found four new bugs in the first week. The find rate was falling because the bench was looking less, not because the design was clean. Plot find rate against regression volume, always.
Walk every code-coverage hole and classify it: a missing plan item (a feature the plan forgot, such as the DMA's unmapped-address error), dead or unreachable code (an exclusion with a reviewed reason), or a configuration that is not built. The plan reaching 100% while the code has holes usually means the plan is incomplete.
Whether the bench can still fail: run a mutation build (a planted bug) and confirm the tests that reach it fail; check waivers and severity demotions added in that period; check the number of compares per test has not dropped; and check the regression volume has not shrunk.
Next: the code-coverage row of this sheet has its own lesson: Code Coverage and How It Closes with Functional Coverage.