A designer fixes a typo in DCRouter's arbiter and commits. The full set of tests from this course (directed, error injection, back-pressure, the random hunts and their Bug Zoo runs, the formal proofs) takes over an hour on a laptop. Nobody waits an hour per commit, so the designer runs nothing, and the next morning's run shows that the typo fix broke round-robin rotation. Three other commits landed overnight; which one did it?
How do you design a regression that catches breakage quickly without taking forever?
One obvious answer is "run everything, always": thorough, and so slow that people stop running it, failures pile up between runs, and finding the commit that broke something becomes its own project. The other, "run a few smoke tests", is fast and misses most breakage. The answer is tiers: different sets of tests at different frequencies, each sized for when it runs.
Tier | When | Budget | Contents |
|---|---|---|---|
Pre-commit or commit check | Every change, before or as it lands | Minutes | Smoke tests, quick directed tests, fast formal checks, lint |
Nightly | Every night | Hours | All directed tests, a few seeds of every random test, all formal proofs |
Weekly or milestone | Weekly, and before milestones | A day or more | Many seeds of every random test, long tests, full coverage merge, mutation or bug-seeding runs |
On demand | When a feature changes | As needed | Deep runs focused on the changed feature |
Tier | DCRouter contents |
|---|---|
Commit check (minutes) | Smoke test (lesson 0.7), directed tests (4.1), the formal proof of FIFO, arbiter and credit properties (7.9, seconds) |
Nightly | Error injection, back-pressure, checked and covered hunts with fresh seeds; merged coverage |
Weekly | Twenty or more seeds of each random test; the full Bug Zoo runs against the main testbenches, as a check that the testbench still catches every planted bug |
The Bug Zoo runs are worth noting: they test the testbench, not the design. A weekly run that suddenly catches 22 of 24 instead of 24 means a check was weakened by some change.
Order tests by value: those that fail most often or cover most, first (Part 12).
Select by change: run more of the tests that exercise what changed.
Kill redundancy: tests that never fail and cover nothing unique are candidates for the weekly tier.
Triage every failure the same day: a failure left a week becomes several overlapping failures.
Treat flaky tests as bugs: a test that fails intermittently is either a testbench bug or a rare design bug (lesson 4.9); never ignore it.
Log seeds and builds: every failure reproducible exactly.
Practice | Purpose |
|---|---|
Continuous integration with tiered regressions | Breakage caught within minutes or hours, with a short list of suspect changes |
Regression ranking and test selection | Most bugs found per simulation hour |
Automatic bisection of failures to a commit | The breaking change found without human search |
Dashboards of failures by first-seen commit and owner | Every failure owned |
A team (illustrative) had a strong nightly regression and no commit check. Each morning brought ten to thirty failures from a dozen overnight commits, and engineers spent their mornings working out whose change caused what. A ten-minute commit check catching the most common breakage would have pointed at a single change each time, before others were stacked on top.
They take a long time (24 builds per testbench) and they check the testbench's strength rather than the design, which changes slowly; a drop in the catch count flags a weakened check.
The bugs it does catch are pinned to a single change immediately, instead of being found overnight among many changes.
Tier the regression: minutes per commit, hours nightly, a day weekly.
Order and select tests by value and by what changed; triage failures the same day.
Treat flaky tests as bugs, log seeds, and include runs that test the testbench itself.