Monday morning: 500 of the weekend's regression runs failed. Each log ends with a different-looking error, because each contains a different time, flit value and port. Opening logs one at a time, an engineer might reach fifty by lunch and still not know whether there are two bugs or forty.
500 tests failed overnight. Where do you start?
The obvious approaches are to start at the top of the list or with whatever looks worst. The top of the list is arbitrary, and "worst-looking" is usually a consequence rather than a cause. Most mass failures come from a handful of causes; the job is to find the handful.
Take the first error of each run: later errors are usually consequences (lesson 13.1).
Abstract it into a signature: replace times, values, ports and counts with placeholders, keeping the words and rule names.
Bucket by signature, largest bucket first.
Debug one representative per bucket: the earliest failure, which has the least noise before it.
Re-bucket after each fix: buckets that disappear were the same bug.
500 failure lines in the formats DCRouter's testbenches print, generated from a few underlying causes with random times, values and ports.
"""Triage at scale: bucketing 500 overnight failures by signature.
500 failing runs (illustrative) print first-error lines in the formats DCRouter's testbenches
use. Underneath, there are only a few causes, in unequal numbers. The raw lines are almost all
different because they contain times, flit values and port numbers. A signature replaces
every number with N, every flit value with H, every bit vector with B, and keeps rule names such as R13; failures with the same signature go in one
bucket. The model prints the buckets, largest first, with one representative each: the run
that failed earliest, which is usually the easiest to debug.
"""
import random
import re
rng = random.Random(5)
templates = [
(0.46, lambda: f"ERROR @{rng.randrange(100, 400000)}: out{rng.randrange(4)}: unexpected flit {rng.randrange(65536):04x}"),
(0.22, lambda: f"ERROR @{rng.randrange(100, 400000)}: credit audit: {rng.randrange(1, 4)} flits left output {rng.randrange(4)} into a stalled sink, expected 4"),
(0.15, lambda: f"ERROR @{rng.randrange(100, 400000)}: out{rng.randrange(4)}: flit {rng.randrange(65536):04x}, expected {rng.randrange(65536):04x} from input {rng.randrange(4)}"),
(0.10, lambda: f"ERROR @{rng.randrange(100, 400000)}: R13: in_vld 1000 did not wake the router"),
(0.05, lambda: f"ERROR @{rng.randrange(100, 400000)}: {rng.randrange(20, 200)} flits still pending after 40000 cycles (lost or stuck)"),
(0.02, lambda: "FAIL: global timeout"),
]
lines = []
for _ in range(500):
u, acc = rng.random(), 0.0
for w, make in templates:
acc += w
if u < acc:
lines.append(make())
break
else:
lines.append(templates[-1][1]())
def signature(line):
s = re.sub(r"(flit|expected) [0-9a-f]{4}\b", r"\1 H", line) # flit values are hex
s = re.sub(r"in_vld [01]{4}", "in_vld B", s) # bit vectors
return re.sub(r"(?<![R\d])\d+", "N", s) # numbers, but keep rule names like R13
buckets = {}
for line in lines:
buckets.setdefault(signature(line), []).append(line)
time_of = lambda l: int(re.search(r"@(\d+)", l).group(1)) if "@" in l else 10**9
print(f"failures: {len(lines)}, distinct raw lines: {len(set(lines))}, buckets: {len(buckets)}")
for sig, members in sorted(buckets.items(), key=lambda kv: -len(kv[1])):
first = min(members, key=time_of)
print(f"{len(members):4d} {sig}")
print(f" first: {first}")
assert len(buckets) <= 8 and len(set(lines)) > 400
print(f"PASS: {len(set(lines))} different failure lines fall into {len(buckets)} signatures; debug one representative per bucket, largest first")It prints:
failures: 500, distinct raw lines: 490, buckets: 6
238 ERROR @N: outN: unexpected flit H
first: ERROR @160: out2: unexpected flit 3d2f
104 ERROR @N: credit audit: N flits left output N into a stalled sink, expected N
first: ERROR @4859: credit audit: 2 flits left output 0 into a stalled sink, expected 4
70 ERROR @N: outN: flit H, expected H from input N
first: ERROR @14054: out3: flit b314, expected b8da from input 1
49 ERROR @N: R13: in_vld B did not wake the router
first: ERROR @931: R13: in_vld 1000 did not wake the router
28 ERROR @N: N flits still pending after N cycles (lost or stuck)
first: ERROR @7625: 97 flits still pending after 40000 cycles (lost or stuck)
11 FAIL: global timeout
first: FAIL: global timeout
PASS: 490 different failure lines fall into 6 signatures; debug one representative per bucket, largest firstHow to read it: 490 of the 500 lines are textually different, yet there are only 6 signatures. Starting with the largest bucket (238 "unexpected flit" failures) and its earliest run at time 160 is the obvious first move. The R13 bucket names its rule outright. The 11 global timeouts carry no information in their first line, so they need the wait-following method (lesson 13.3), or they will turn out to be consequences of another bucket's bug.
Practice | Why |
|---|---|
Failure messages with stable wording and named rules | Signatures group by cause, not by phrasing |
Abstract numbers, keep identifiers that matter | R13 kept; times and flit values removed (port numbers can be kept or dropped by choice) |
First error only | Cascades of consequences do not create false buckets |
Link buckets to known bugs | Known failures do not get debugged twice |
Practice | Purpose |
|---|---|
Automatic signature bucketing in regression dashboards | Mass failures summarised as a few causes |
Buckets linked to bug tickets and first-failing commits | Owners and suspects assigned automatically |
Rerun of one representative per bucket with waves and debug enabled | Debug data ready when the engineer arrives |
A team (illustrative) bucketed failures by the full error text. Every failure with a different timestamp became its own bucket, and the dashboard showed 480 "unique" failures from what turned out to be three bugs. Abstracting numbers out of the signature collapsed them to the three causes in a minute.
R13 names the rule that failed, which identifies the cause; times and values differ between runs of the same bug.
It has the least activity before the failure, so its trace is shortest and least confused by consequences.
Mass failures usually have a few causes: find them by signature.
Use the first error, abstract numbers, keep rule names, bucket and rank by size.
Debug one representative per bucket, the earliest, and re-bucket after each fix.