One test failing is a debugging problem. Three hundred tests failing overnight is a management problem: which failures are the same bug, which are new, who owns each one, and is the design getting better or worse? This topic is about running a bench at scale and turning the result into decisions, from the seed that reproduces one failure to the review where a block is signed off.
Reproduce any regression failure exactly from a recorded bundle of test, seed, build and plusargs, and tell a real failure from a flaky one.
Bucket a night of failures by first-error signature with a short script, and debug one run per bucket instead of every run.
Design a regression flow: one build, many runs, tiered test lists, a farm, and a CI gate on merges.
Get waveforms for a failure without dumping the whole regression.
Present the numbers a sign-off review asks for, and know which ones can lie.
Lesson | The question it answers |
|---|---|
Why does a failure not come back when I rerun it, and what must I record so it always does? | |
Thirty runs failed: how many bugs is that, and which run do I debug first? | |
How do you run thousands of tests a night without rebuilding thousands of times, and what blocks a merge? | |
How do you get the waves for one failure without dumping terabytes? | |
What numbers prove a block is done, and how do they mislead? |