Labels

Engineers label past nights once, and every agent version is graded against the same labels. Each label records the true group per run and the true culprit commit per group.

Cluster metrics

Clustering is graded as pairs. Precision asks how many proposed pairs are true pairs, and recall asks how many true pairs were proposed.

1
2
3
4

proposed pairs = 40, correct = 30
precision = 30 / 40 = 0.75 = 75%
true pairs = 50, found = 30
recall = 30 / 50 = 0.6 = 60%

Suspect metrics

Ranking is graded by top-1 accuracy. The check is whether the first suspect on the card matches the true culprit commit.

1
2
3
4
5
6
7

50 labelled nights, top suspect correct on 34
top-1 accuracy = 34 / 50 = 0.68 = 68%
outcome split over the 50 nights:
top-1 correct: 34 = 68%
correct suspect in top 3 only: 9 = 18%
culprit missed: 7 = 14%
check: 68 + 18 + 14 = 100

Operating the loop

Cost is tracked per night alongside quality. Token prices below are a stated example parameter, not any real price.

assume 1 million input tokens cost 3 units (example parameter, not a real price)
one nightly pass reads 200000 tokens
cost = 200000 / 1000000 x 3 = 0.2 x 3 = 0.6 units per night

Related topics

• The nightly triage loop

• Evaluating AI tools for your team