The dashboard request

The programme director wants a debug dashboard by the end of the month. The draft has three tiles: bugs closed per engineer per week, open bug count, and regression pass rate, target 95% for tape-out. You are asked to sign it off as the debug lead.

Which of these will make the team faster, which will make it look faster, and what would you put in their place?

What people usually do, and why it fails

Dashboards are built from what the tracker and the regression already count, and counts are easy to move. Once a number becomes a target, people meet the target by the cheapest route, which is rarely finding bugs. It is an old observation that a measure which becomes a target stops measuring well, and in debugging it takes precise, predictable forms:

None of these needs bad faith. Each is the rational response of a busy engineer to a number their review depends on.

The method: measure the flow, read the tail

A bug moves through the states of the tracker. The useful metrics are the times between those states, because each interval belongs to one step of the debug loop and tells you which step is slow:

1. Medians and 90th percentiles, never means alone. A few long bugs dominate a mean. The median says how a normal bug goes; the 90th percentile says how bad the bad ones are, and that is where the cost is.

2. Split by bug class and stage, not by person. "Hangs found on the bench take 120 hours to reproduce" is actionable: build a hang detector. "Engineer A is slow" is not, and it teaches A to avoid hard bugs.

3. Pair speed with quality. Report time to close next to reopen rate, and closures next to closed as fixed and never reopened.

4. Look for the step, then fix the process. A long time to reproduce is a reason to invest in replay records, not to tell people to reproduce faster.

The picture

Do it: a pass-rate target, met with five bugs left

dbglab at HEAD has five known bugs. Here is its regression with the default seeds:

$ tb/regress.sh icarus 1 2 3
...
irq_w1c                  seed 1      FAIL
...
dma_burst_backpressure   seed 1      FAIL
...
periph_power             seed 3      FAIL
random                   seed 1      PASS
random                   seed 2      PASS
random                   seed 3      PASS
9 failing run(s)

Nine of 21 runs fail: a 57% pass rate, far below a 95% target. The three failing tests each have a tracked bug. The cheap way to the target is to waive them as known failures:

$ n=0; f=0; for t in smoke dma_copy periph_stream random; do for s in 1 2 3; do n=$((n+1)); tb/run.sh $t $s > /dev/null || f=$((f+1)); done; done; echo "$f failing of $n"
0 failing of 12
$ for s in $(seq 1 60); do tb/run.sh random $s | grep FAIL; done | wc -l
      19

100% pass. The dashboard turns green, and the same design fails random on 19 of 60 seeds, from two bugs that the default seeds never reach. Nobody lied: the waived tests are tracked, and seeds 1 to 3 are the documented default. The target measured the choice of what to run, not the state of the design. A useful version of the tile shows the pass rate and the number of tracked S1 and S2 bugs, and runs a seed sweep wide enough to reach rare paths (how wide is a calculation: statistics of rare failures).

Do it: flow metrics from a tracker export

The calculator below reads a tracker export and prints the flow metrics and two rankings. The twelve tickets are illustrative, made up for this page to show typical shapes; the arithmetic is what a real export would get. It checks its own results and prints PASS.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

"""metrics.py: debug metrics from a bug tracker export. The twelve tickets below are ILLUSTRATIVE,
made up for this page; the calculations are what a real export would get.
Hours are counted from when the ticket was opened. None means the step never happened."""
import math
import statistics

# id, owner, resolution, hours to reproduce, hours to root cause, hours to close, reopened later
TICKETS = [
    ("T1", "A", "fixed", 2, 6, 8, False),
    ("T2", "A", "duplicate", None, None, 1, False),
    ("T3", "A", "cannot-reproduce", None, None, 3, True),
    ("T4", "A", "fixed", 1, 2, 3, True),          # the symptom was patched; it came back
    ("T5", "A", "duplicate", None, None, 1, False),
    ("T6", "A", "cannot-reproduce", None, None, 2, False),
    ("T7", "B", "fixed", 4, 20, 30, False),
    ("T8", "B", "fixed", 3, 12, 16, False),
    ("T9", "B", "fixed", 30, 70, 80, False),      # an intermittent: most of the time was reproduction
    ("T10", "C", "fixed", 1, 3, 5, False),
    ("T11", "C", "fixed", 2, 5, 7, False),
    ("T12", "C", "fixed", 120, 160, 170, False),  # a hang seen first on the bench
]

def p90(xs):
    """Nearest-rank 90th percentile."""
    s = sorted(xs)
    return s[math.ceil(0.9 * len(s)) - 1]

ttr = [t[3] for t in TICKETS if t[3] is not None]
ttrc = [t[4] for t in TICKETS if t[4] is not None]
print(f"time to reproduce   n={len(ttr)}  median={statistics.median(ttr):5.1f} h  mean={statistics.mean(ttr):5.1f} h  p90={p90(ttr)} h")
print(f"time to root cause  n={len(ttrc)}  median={statistics.median(ttrc):5.1f} h  mean={statistics.mean(ttrc):5.1f} h  p90={p90(ttrc)} h")
share = [t[3] / t[4] for t in TICKETS if t[3] is not None]
print(f"reproduction's share of time to root cause: median {statistics.median(share):.0%}")

print("\nowner  closed  closed as fixed and never reopened  reopened")
rank = {}
for o in "ABC":
    mine = [t for t in TICKETS if t[1] == o]
    good = [t for t in mine if t[2] == "fixed" and not t[6]]
    back = [t for t in mine if t[6]]
    rank[o] = (len(mine), len(good))
    print(f"  {o}      {len(mine)}       {len(good)}                                    {len(back)}")
reopen = sum(t[6] for t in TICKETS) / len(TICKETS)
cnr = [t for t in TICKETS if t[2] == "cannot-reproduce"]
print(f"\nreopen rate {reopen:.0%}; closed as cannot-reproduce: {len(cnr)}, reopened among them: {sum(t[6] for t in cnr)}")

# self-check: the numbers the page quotes
assert (statistics.median(ttr), statistics.mean(ttr), p90(ttr)) == (2.5, 20.375, 120)
assert (statistics.median(ttrc), p90(ttrc)) == (9.0, 160)
assert max(rank, key=lambda o: rank[o][0]) == "A" and rank["A"][1] == 1 and rank["B"][1] == 3
print("PASS: medians, tails and the two rankings as quoted")

$ python3 -I metrics.py
time to reproduce   n=8  median=  2.5 h  mean= 20.4 h  p90=120 h
time to root cause  n=8  median=  9.0 h  mean= 34.8 h  p90=160 h
reproduction's share of time to root cause: median 37%

owner  closed  closed as fixed and never reopened  reopened
  A      6       1                                    2
  B      3       3                                    0
  C      3       3                                    0

reopen rate 17%; closed as cannot-reproduce: 2, reopened among them: 1
PASS: medians, tails and the two rankings as quoted

Read the first two lines as a debug lead would. The median bug reproduces in two and a half hours, but the mean is eight times that, pulled up by two tickets: the intermittent (30 hours) and the bench hang (120 hours). The 90th percentile, 120 hours, names the problem the mean hides. Reproduction takes a little over a third of the time to root cause for the median bug and three quarters of it for the bench hang. That points at investment in capture and replay for hangs, not at the team's speed.

Now the rankings. By closed, A leads with six. By closed as fixed and never reopened, A has one and B and C have three each. Two of A's closures were duplicates, two were "cannot reproduce" (one came back), and one fix patched a symptom and reopened. A closure count would have praised exactly the behaviour that costs the team later.

What the evidence says

The two dbglab runs show how a target on a count can be met with no change to the design, and the calculator shows how a mean and a closure count hide the expensive part of the work. The useful numbers were the interval medians, their tails, and closures paired with reopenings. They point at a process step to improve rather than at a person to push.

Traps

Related topics

• Part 11 · Debug as a Team

• Bug tracking and severity

• The cost of a bug by stage

• Statistics of rare failures

• Part 9: fix, prove, prevent