Formal Coverage Metrics: Beyond Green Proof Counts: Expanded Case Study

Extended case study

A formal regression involving Formal Coverage Metrics: Beyond Green Proof Counts reopens late in the release cycle after RTL and constraint updates.

Background

Earlier runs were stable, but model assumptions drifted and property intent was not re-audited after implementation changes.

Symptoms observed

• non-vacuous closure rate, counterexample turnaround time, and requirement-level residual risk trend trends worsen while status dashboards look superficially stable.

• counterexample patterns recur across related properties.

• reviewers disagree on whether failures are real bugs or modeling artifacts.

Investigation timeline

1. Hour 0: freeze RTL, assumptions, and tool settings for reproducibility.

2. Hour 1: classify failures into bug, model mismatch, or weak-property buckets.

3. Hour 2: isolate first divergence and map to requirement intent.

4. Hour 3: apply one constrained change and rerun focused property set.

5. Hour 4: confirm reachability and vacuity quality did not regress.

6. Hour 5: replay representative traces in simulation or equivalent flow.

7. Hour 6: publish closure memo with residual risk classification.

Root cause

Root cause traced to Formal Coverage Metrics: Beyond Green Proof Counts: Formal closure quality depends on metric meaning, not property pass volume.

Fix and validation

• Correct assumption/property scope to preserve legal behavior.

• Add targeted helper checks that expose key intermediate invariants.

• Update runbook and requirement traceability for future regression stability.

Lessons learned

• Status color is not proof quality; audit supporting evidence.

• First-divergence classification outperforms broad trace inspection.

• Constraint and abstraction governance must be versioned and reviewed.

CASE STUDY - Formal Coverage Metrics: Beyond Green Proof Counts
closure slope / vacuity trend / inconclusive aging / replay confidence

Formal deep dive

Signoff quality is requirement-centric and must integrate proof status, reachability, bounded limits, and waiver governance.

Concept diagram

FORMAL SIGNOFF PYRAMID

requirements -> properties and covers -> quality metrics -> waiver governance -> release decision

Metric graph

SIGNOFF CONFIDENCE TREND

fully proven critical    ███████
bounded-only critical    ████
unexplained cover gaps   ███

Metrics and artifacts to collect

• requirement-to-proof closure map

• critical cover reachability and gap aging

• bounded-only risk register

• waiver debt with owner and expiry

Mini case study

A release review blocked signoff until bounded-only properties were paired with explicit residual-risk and replay plans.

Debug branches

• Separate status color from proof quality dimensions.

• Treat unreachable critical covers as signoff blockers.

• Document bounded-horizon rationale with architecture limits.

Senior review question

Ask: which requirement intent is proven, under which assumptions, and what residual risk remains?

Key takeaways

• Tie each proof claim to assumption boundaries and reachability evidence.

• Prefer minimal reversible fixes and preserve legal behavior visibility.

Common pitfalls

• Treating runtime reduction as proof-quality improvement without audits.

• Declaring closure while critical covers remain unreachable.

• Using broad waivers instead of first-divergence root-cause ownership.

Principal formal review addendum

Formal Coverage Metrics: Beyond Green Proof Counts should be reviewed as a requirement-evidence workflow, not a single status report.

Use non-vacuous closure rate, counterexample turnaround time, and requirement-level residual risk trend as the monitoring lens and formal closure packet: assumptions audit, proof status matrix, counterexample classification, and requirement traceability as closure proof.

Signoff is requirement-centric evidence synthesis, not a single dashboard percentage. Strong teams preserve legal reachability while improving convergence.