A weekly status report shows a regression pass rate rising steadily: 21%, 42%, 64%, 85%, and finally 100% for three weeks running. Management is pleased. The verification lead is not, because the same report shows that the share of features any test has ever reached has been stuck at two thirds for the whole quarter. The pass rate measures agreement between the tests and the design. It says nothing about the features no test looks at.

Which metrics can you trust, and what does each hide?

The obvious answer, and why it breaks

The obvious metric is the one that is easiest to report: pass rate, or a single coverage percentage. Each measures one narrow thing, and each can be improved without improving verification: by fixing tests rather than bugs, by adding easy tests, by excluding hard bins, by not looking. Goodhart's warning applies: a metric that becomes a target stops measuring what it was chosen for.

The idea: what each metric measures, and how it misleads

Build it: a pass rate that lies

A model of twelve weeks on a 30-feature design. The team fixes the bugs its tests expose and writes 40 new tests a week, mostly where writing tests is easy, so new tests rarely reach an untested feature.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

"""Metrics and their traps: a pass rate that climbs while the risk stays.

A design has 30 features; 18 have been reached by some test, 12 have not (illustrative).
Hidden bugs: 2 in each reached feature, 1 in each unreached one. Each week:
  - the team fixes the bugs that currently fail tests (up to 4 a week);
  - it adds 40 new tests, but new tests are written where it is easy, so each one reaches an
    unreached feature with only a 0.5% chance (otherwise it repeats a reached feature);
  - a test fails if its feature still holds an unfixed bug.
The model prints, per week, the pass rate, the share of features reached, and the bugs left.
"""
import random

rng = random.Random(8)
F = 30
reached = set(range(18))
bugs = {f: (2 if f in reached else 1) for f in range(F)}
tests = [rng.choice(sorted(reached)) for _ in range(60)]
print("week  tests  pass%  features reached%  bugs left (in unreached features)")
rows = []
for week in range(1, 13):
    failing = sorted({f for f in tests if bugs[f] > 0})
    for f in failing[:4]:
        bugs[f] -= 1
    for _ in range(40):
        if rng.random() < 0.005 and len(reached) < F:
            f = rng.choice([x for x in range(F) if x not in reached])
            reached.add(f)
        else:
            f = rng.choice(sorted(reached))
        tests.append(f)
    passing = sum(1 for f in tests if bugs[f] == 0)
    left = sum(bugs.values())
    hidden = sum(bugs[f] for f in range(F) if f not in reached)
    rows.append((week, len(tests), 100 * passing / len(tests), 100 * len(reached) / F, left, hidden))
    print(f"{week:4d}  {len(tests):5d}  {100 * passing / len(tests):5.1f}  {100 * len(reached) / F:17.0f}  {left:3d} ({hidden})")

last = rows[-1]
assert last[2] > 97 and last[3] < 80 and last[5] >= 5
print(f"PASS: the pass rate reaches {last[2]:.1f}% while {100 - last[3]:.0f}% of features have never been tested and {last[5]} bugs sit in them")

It prints:

week  tests  pass%  features reached%  bugs left (in unreached features)
   1    100    0.0                 63   44 (11)
   2    140   21.4                 63   40 (11)
   3    180   23.3                 63   36 (11)
   4    220   42.3                 63   32 (11)
   5    260   43.1                 63   28 (11)
   6    300   64.0                 63   24 (11)
   7    340   63.5                 63   20 (11)
   8    380   85.3                 67   16 (10)
   9    420   89.5                 67   12 (10)
  10    460  100.0                 67   10 (10)
  11    500  100.0                 67   10 (10)
  12    540  100.0                 67   10 (10)
PASS: the pass rate reaches 100.0% while 33% of features have never been tested and 10 bugs sit in them

How to read it: the pass rate rises because the team fixes the bugs its tests can see; it reaches 100% in week 10 and stays there. Features reached barely moves, from 63% to 67%, because new tests go where tests already are. At the end, 10 bugs remain and every one of them is in a feature no test has ever touched. The pass rate is accurate and irrelevant to the question that matters.

Reading metrics well

• Read metrics together: pass rate with feature reach and coverage; bug rate with effort.

• Read them against the plan: the share of planned features with evidence is the metric that answers "how verified".

• Look at holes and trends, not single numbers: what is missing, and what is changing.

• Watch for gaming: excluded bins, disabled tests and easy tests all move metrics without moving risk.

How real teams use metrics

What breaks

The model is the story. In real projects the same pattern appears as a green regression dashboard and a late surprise: a configuration, an error path or a power mode nobody tested, discovered at integration or in the lab. A plan-based metric ("20 of 30 features have evidence") would have shown the gap every week.

Key takeaways

• Every metric measures something narrower than how verified the design is.

• A pass rate can reach 100% while features remain untested and bugs remain in them.

• Read metrics together and against the plan; report exclusions beside them.

Related topics

• Part 9 overview: Planning and Risk