Each suspect commit carries three evidence bits. File overlap asks whether it touched the failing block, recency asks whether it is new, and bisect asks whether reruns blame it.
Evidence | Question answered | Source |
|---|---|---|
File overlap | Did the commit touch the failing block? | Changed-file list. |
Recency | Is the commit within the last two days? | Commit dates. |
Bisect vote | Does the range search blame it? | Rerun results. |
A weighted sum turns the three bits into one order. The weights below are stated example parameters, and every score is shown with its substitution.
score = 4 x overlap + 2 x recent + 3 x bisect
c1: 4 x 1 + 2 x 1 + 3 x 1 = 4 + 2 + 3 = 9
c3: 4 x 1 + 2 x 0 + 3 x 0 = 4 + 0 + 0 = 4
c2: 4 x 0 + 2 x 1 + 3 x 0 = 0 + 2 + 0 = 2
ranked order: c1 (9), c3 (4), c2 (2)A passing rerun does not clear a suspect. Flaky cases pass on their own, so the agent needs several clean reruns before downgrading a suspect.
assume one rerun passes a flaky case with probability 0.3 (stated example parameter)
three reruns all pass: 0.3^3 = 0.3 x 0.3 x 0.3 = 0.027 = 2.7%
even a flaky case that passes 30% of the time passes three reruns in a row 2.7% of the time, so passing reruns reduce doubt but do not prove the fixThe program below is a scripted stand-in for the ranking step, not a model. It scores the three suspects above and checks the order c1, c3, c2 with scores 9, 4 and 2.
import sys
SUSPECTS = [
{"id": "c1", "overlap": 1, "recent": 1, "bisect": 1},
{"id": "c2", "overlap": 0, "recent": 1, "bisect": 0},
{"id": "c3", "overlap": 1, "recent": 0, "bisect": 0},
]
W_OVERLAP = 4
W_RECENT = 2
W_BISECT = 3
def score(s):
return W_OVERLAP * s["overlap"] + W_RECENT * s["recent"] + W_BISECT * s["bisect"]
ranked = sorted(SUSPECTS, key=score, reverse=True)
order = [s["id"] for s in ranked]
scores = [score(s) for s in ranked]
if order != ["c1", "c3", "c2"]:
print("FAIL: wrong order")
sys.exit(1)
if scores != [9, 4, 2]:
print("FAIL: wrong scores")
sys.exit(1)
print("PASS: ranked order is c1, c3, c2 with scores 9, 4 and 2")It touches none of the failing files and bisect clears it. Recency alone scores 2 against 9 and 4.
Not on reruns alone. A flaky case that passes 30% of the time still passes three in a row 2.7% of the time, so file and bisect evidence still count.