Part 11 · Senior Prep · Intermediate

Regression Sign-Off Interview Q&A

Model answers on regression tiers, sign-off metrics dashboards, waiver audits, trend analysis, and presenting evidence to leads.

Regression structure and sign-off metrics

Q: How do you tier regressions — smoke, nightly, weekly?

diagram

[INT][SENIOR][UVM] MODEL ANSWER

Q: Regression tiers?

A:
  MECHANISM:  Smoke (minutes, every commit), nightly (hours, broad random seeds),
              weekly (full plan + closure + negative suite).
  MOTIVATION:  Fast feedback on commits; depth before milestone gates.
  SMOKE:      P0 tests, 1–3 seeds, assert no UVM_ERROR/FATAL.
  NIGHTLY:    random stress + merge cov + failure manifest.
  PITFALL:    Only nightly exists — developers wait 8h for basic break detection.
  EXAMPLE:    Pre-commit smoke 12 tests; nightly 80 tests × 50 seeds; weekly adds formal.

Q: What goes on a sign-off metrics dashboard?

diagram

[INT][SENIOR][UVM] MODEL ANSWER

Q: Sign-off dashboard contents?

A:
  MECHANISM:  Merged functional coverage by plan section, P0 hole count, waiver
              list, assertion status, 7-day trend, open bug severity histogram.
  MOTIVATION:  Leads decide tape-out risk from evidence package — not verbal status.
  METRICS:    func cov % per plan feature, P0/P1 holes, waiver audit, bug P0 count.
  PITFALL:    Code coverage alone on dashboard — wrong metric for plan compliance.
  EXAMPLE:    "AXI 98.2% plan cov, 1 P0 hole (closure Thu), 2 approved waivers."

diagram

[INT][SENIOR][UVM] sign-off scorecard template

  PLAN SECTION     MERGED FUNC COV    P0 HOLES    WAIVERS
  AXI protocol     98.2%              0           0
  DMA paths        94.1%              1 (FP-12)   0
  Interrupts       91.0%              2           1 (approved)
  Reset corners    100%               0           0

  TREND (7d): DMA +3.2%, IRQ +1.1%
  ASSERTIONS: 142/142 passing on nightly merge

Key takeaways

Three regression tiers — smoke for speed, nightly for depth, weekly for gates.
Dashboard = plan func cov + P0 holes + waivers + trend — not code cov alone.
Present deltas week-over-week — shows closure velocity.

Common pitfalls

Green regression with flat merged coverage — no new bin progress.
Dashboard without waiver audit — tape-out review fails.

Presenting sign-off evidence

Q: Design lead challenges a waived bin — you respond?

diagram

[INT][SENIOR][UVM] MODEL ANSWER

Q: Challenge to waiver?

A:
  MECHANISM:  Produce spec citation, alternative coverage proof, or repro showing
              unreachable with design concurrence on record.
  MOTIVATION:  Waivers are liability — must survive adversarial review.
  RESPONSE:   Show waiver form: plan ID, bin, errata cite, approver email, date.
              Offer directed repro if they believe reachable.
  PITFALL:    Defensive argument without data — erodes trust.
  EXAMPLE:    FP-33 waived per errata C.2.1 — feature removed; show spec diff PDF.

Q: How do you know regression is healthy vs noisy?

diagram

[INT][SENIOR][UVM] MODEL ANSWER

Q: Healthy vs noisy regression?

A:
  MECHANISM:  Track failure rate, flake rate (pass on rerun same seed), new failure
              delta vs last green baseline.
  MOTIVATION:  Noisy farm erodes trust — team ignores reds.
  HEALTHY:    <2% flake, failures cluster to one check-in, owners assigned <24h.
  NOISY:      Same test fails/passes alternate seeds — fix seed stability or checker.
  PITFALL:    Disabling failing test instead of fixing flake — hides real bugs.
  EXAMPLE:    axi_rand 3% flake from race in scoreboard — fix sync, not remove test.

bash

# compare tonight vs last green manifest
diff <(sort last_green.manifest) <(sort tonight.manifest) > new_failures.txt
head -20 new_failures.txt
# flake check: rerun same seed 5x
for i in 1 2 3 4 5; do
  simv +UVM_TESTNAME=axi_rand_test +ntb_random_seed=4099 -l flake_${i}.log
done
grep -c UVM_ERROR flake_*.log

Q: Formal vs simulation sign-off — how do they complement?

diagram

[INT][SENIOR][UVM] MODEL ANSWER

Q: Formal + sim sign-off?

A:
  MECHANISM:  Formal proves properties on control paths; sim proves data path scenarios
              and software-visible sequences formal cannot reach easily.
  MOTIVATION:  Neither alone closes a complex SoC — complementary evidence.
  WHEN:       Formal for arbiters, FIFO bounds, one-hot; sim for system scenarios.
  PITFALL:    Claiming formal replaces UVM — data transforms need sim checkers.
  EXAMPLE:    Formal proves grant fairness; sim proves DMA+CPU interleave with SB.

Key takeaways

Waiver defense = spec cite + approver audit + offer repro.
Track flake rate — noisy regression trains team to ignore failures.
Formal and sim are complementary — cite both on dashboard when available.

Common pitfalls

Removing flaky tests from nightly — hole in regression permanently.
Sign-off presentation without trend — leads cannot judge velocity.

Practice this lessonQuestions tagged for this topic in the bank