Metrics fail in predictable ways, and each failure flatters the project. This lesson names the three traps that appear on every programme, then states the comparison rules that make trends trustworthy. Honesty here is procedural: rules enforced by the dashboard, not virtues hoped for in the team.
Trap | How it flatters | The rule that stops it |
|---|---|---|
Counting waived as fixed | Permissions inflate the fix rate; risk hides inside “closed” | Fixed and waived are separate series forever; “closed” is never a headline number |
Hiding hierarchies | Excluded blocks vanish from counts; the denominator shrinks silently | Every run reports its scope: hierarchy included, excluded regions with owners, and block revisions |
Comparing runs with different setup | A setup narrowing looks like a fix avalanche | Trend lines break at setup changes; pre- and post-change runs are never joined into one slope |
Same setup version, same scope, same block revisions — or the comparison is annotated as broken and no slope is drawn.
New findings and fixed findings are reported separately, so churn never disguises stagnation.
Waivers appear in their own series with age and revision currency; a rising waiver count alongside falling defects triggers review, not applause.
Constraint changes reset the comparison for the affected crossings: intent moved, so history restarts there.
These rules feel bureaucratic until the first milestone dispute, when they become the only shared facts in the room. A dashboard that enforces them quietly, every run, is worth more than a heroic audit the night before tape-out.
A broken trend: the two numbers side by side with no joining slope, annotated with the setup change. The drop is uninterpretable until a same-setup re-run confirms it.
Hidden hierarchy. The fix is to name the excluded region, its model, and its owner on every run — exclusion with accountability, not silence.