The agent needs its own regression suite, separate from the chip regression it operates. It pins agent behavior across prompt, tool, and permission changes.
Each case fixes the input, the allowed tools, and the expected outcome. The suite in the table below uses 60 pinned triage cases.
Area | Cases | Example expectation |
|---|---|---|
Triage verdicts | 25 | Culprit test named, confirmed by rerun. |
Lint fixes | 20 | Fix applies, full lint clean, no new warnings. |
Refusals and escalation | 15 | Secrets in logs trigger refusal plus escalation. |
Run the full suite before and after each change to prompts, tools, models, or permissions. Record both versions with the result so runs stay comparable.
Small prompt edits can move several cases at once. The suite exists because case-by-case inspection by hand misses those moves.
Compare case by case, not just totals. The calc block below works an EXAMPLE change from 48 to 51 of 60.
suite: n = 60 pinned cases
before: 48 / 60 = 0.80; after: 51 / 60 = 0.85
previously passing now wrong: 2 -> suite gate trips (gate allows 0)
newly passing: 51 - 48 + 2 = 5
net gain +3 hides 2 regressions: totals alone would pass, the gate does notThe program below recomputes the suite comparison from EXAMPLE pass lists. It uses plain lists, with no model involved.
import sys
before = [True] * 48 + [False] * 12
after = [True] * 46 + [False] * 2 + [True] * 5 + [False] * 7
passed_before = sum(1 for b in before if b)
passed_after = sum(1 for a in after if a)
newly_wrong = sum(1 for b, a in zip(before, after) if b and not a)
newly_fixed = sum(1 for b, a in zip(before, after) if not b and a)
gate_trips = newly_wrong > 0
if passed_before != 48:
print("FAIL: before count arithmetic off")
sys.exit(1)
if passed_after != 51:
print("FAIL: after count arithmetic off")
sys.exit(1)
if newly_wrong != 2:
print("FAIL: regression count arithmetic off")
sys.exit(1)
if newly_fixed != 5:
print("FAIL: fix count arithmetic off")
sys.exit(1)
if not gate_trips:
print("FAIL: gate should trip")
sys.exit(1)
print("PASS: suite moves 48 to 51 of 60 with 5 new passes and 2 regressions flagged")5. Newly fixed equals 51 minus 48 plus the 2 that broke.
Totals hide direction. Two working cases broke, so the change regressed behavior the team already relied on.