A block has thirty features and twelve weeks of verification. Every feature could use more effort than the schedule allows. One engineer proposes working through the plan top to bottom, in specification order; another proposes doing the easy features first to show progress. Either way, when the schedule ends, the features left unfinished are decided by accident. Prioritisation decides them on purpose.

How do you decide what to verify first when time is limited?

The obvious answer, and why it breaks

The obvious orders are specification order (simple, arbitrary) and easiest first (visible progress, wrong features). Both spend early effort where bugs are rare and leave the riskiest features for the end, when there is least time and the most pressure to call them done. The order should follow where bugs are likely and expensive.

The idea: likelihood × impact

A simple score such as likelihood × impact (each 1 to 5), adjusted up for features that are hard to detect, gives an order. The exact numbers matter less than doing it explicitly and reviewing it with designers, who know where the design is fragile.

Build it: does risk-weighted effort pay off?

Risk estimates are noisy, often off by a factor of two. Does spending effort in proportion to an imperfect estimate still help? The model simulates 2,000 projects of 20 features with true risks hidden from the plan, and compares three ways of spending the same budget.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

"""Risk-based prioritisation: does spending effort by estimated risk find more bugs?

A project has 20 features. Each has a true risk r (how many bugs it tends to hold); the plan
only knows an estimate, the true risk times a noisy factor (estimates are often off by 2x).
Bugs per feature are drawn with mean proportional to r. A fixed verification budget is split
across features; each bug in a feature is found with probability 1 - exp(-effort / 3).
Three plans spend the same total effort:
  uniform   the same effort on every feature
  weighted  effort proportional to the estimated risk
  top-5     all the effort on the 5 features estimated riskiest
2,000 random projects (illustrative). The model prints the average share of bugs each plan
finds and how often weighted beats uniform.
"""
import math
import random

rng = random.Random(12)
N, BUDGET, PROJECTS = 20, 40.0, 2000

def poisson(lam):
    k, p, l = 0, 1.0, math.exp(-lam)
    while True:
        p *= rng.random()
        if p < l:
            return k
        k += 1

totals = {"uniform": 0.0, "weighted": 0.0, "top-5": 0.0}
wins = 0
for _ in range(PROJECTS):
    r = [rng.lognormvariate(0, 0.8) for _ in range(N)]
    est = [x * rng.lognormvariate(0, 0.7) for x in r]           # estimates off by about 2x
    bugs = [poisson(1.5 * x) for x in r]
    total = sum(bugs) or 1
    top = sorted(range(N), key=lambda i: -est[i])[:5]
    plans = {
        "uniform": [BUDGET / N] * N,
        "weighted": [BUDGET * e / sum(est) for e in est],
        "top-5": [BUDGET / 5 if i in top else 0.0 for i in range(N)],
    }
    found = {k: sum(b * (1 - math.exp(-e / 3)) for b, e in zip(bugs, v)) / total for k, v in plans.items()}
    for k in totals:
        totals[k] += found[k] / PROJECTS
    wins += found["weighted"] > found["uniform"]

for k, v in totals.items():
    print(f"{k:9s} finds {100 * v:5.1f}% of the bugs on average")
print(f"weighted beats uniform in {100 * wins / PROJECTS:.0f}% of projects")
assert totals["weighted"] > totals["uniform"] and totals["weighted"] > totals["top-5"]
print("PASS: spending by estimated risk finds more bugs than spreading evenly, and more than betting everything on the top 5")

It prints:

uniform   finds  48.7% of the bugs on average
weighted  finds  53.7% of the bugs on average
top-5     finds  42.6% of the bugs on average
weighted beats uniform in 79% of projects
PASS: spending by estimated risk finds more bugs than spreading evenly, and more than betting everything on the top 5

How to read it: even with estimates off by about 2x, spending in proportion to estimated risk finds 53.7% of the bugs against 48.7% for an even spread, and wins in 79% of projects. Betting everything on the five features estimated riskiest finds only 42.6%: the estimates are wrong often enough that features left at zero effort hold many bugs. Weight by risk, but keep some effort everywhere.

Using risk in a plan

• Score every feature, with designers, using likelihood, impact and difficulty of detection.

• Schedule high-risk features first, so they get the most calendar time and the earliest feedback.

• Give every feature some effort: a minimum check and coverage goal, because estimates are wrong.

• Re-score as you learn: a feature where bugs keep appearing is riskier than estimated (Part 12).

How real teams prioritise

What breaks

A team (illustrative) ranked features by risk, then spent almost all its time on the top three and gave the rest a smoke test each. A "low-risk" register block, untouched since a previous project, had been re-parameterised for this one; its reset values were wrong for the new width. The model's top-5 row is this team: estimates are imperfect, and features with zero effort hide the bugs the estimate missed.

Key takeaways

• Order work by risk: likelihood of bugs, impact of a bug, and difficulty of detection.

• Even noisy risk estimates help: weight effort by risk.

• Keep a floor of effort on every feature, and re-score as bugs are found.

Related topics

• Part 9 overview: Planning and Risk