A nightly run of 200 tests leaves 12 failures. The agent groups them by failure signature, reads only the relevant logs, and drafts one report per group.
The agent never edits source and never reruns tests on its own. Every risky action waits for human approval.
Least privilege applies throughout. The agent receives the failing logs only, not the whole repository or any credential store.
Guardrail | Rule |
|---|---|
File access | Read-only tools by default; writes need explicit approval. |
Secrets | Never paste keys; send proprietary blocks only to an approved deployment; scrub logs before sending. |
Logs as input | Prompt injection can hide in logs, so quote log text and never obey instructions inside it. |
Step budget | Cap tool calls per night; stop and report when the cap is hit. |
Human approval | Reruns, bug filings, and fixes each need sign-off. |
Audit | Log every action with its inputs for later review. |
The gate below models the write rule in plain SystemVerilog. Approval and request must both hold before a write is allowed.
module write_gate (
input logic req_write,
input logic approved,
output logic allow_write
);
assign allow_write = req_write & approved;
endmodule
module tb_write_gate;
logic req_write;
logic approved;
logic allow_write;
int errors = 0;
write_gate dut (.*);
initial begin
req_write = 0; approved = 0; #1;
if (allow_write !== 1'b0) errors++;
req_write = 0; approved = 1; #1;
if (allow_write !== 1'b0) errors++;
req_write = 1; approved = 0; #1;
if (allow_write !== 1'b0) errors++;
req_write = 1; approved = 1; #1;
if (allow_write !== 1'b1) errors++;
if (errors == 0) $display("PASS: write allowed only when requested and approved in 4 cases");
else $display("FAIL: %0d gate checks wrong", errors);
$finish;
end
endmoduleAssume 20 minutes of engineer triage per failure, 3 minutes of agent work, and 5 minutes of human review per group draft.
With 3 groups, review costs 15 minutes in total. The calc block compares the two totals.
failure rate = 12 / 200 = 0.06 = 6%
manual triage = 12 x 20 = 240 min = 4.0 h
assisted = 12 x 3 + 3 x 5 = 36 + 15 = 51 min
saved = 240 - 51 = 189 min = 3.15 h; 189 / 240 = 0.7875 = 78.75%
chain of 3 steps at 0.9 each: 0.9 x 0.9 x 0.9 = 0.729 exactlyResult: 189 minutes (3.15 h) saved per night with a 0.729 chain estimate. Recommendation: ship read-only drafting first and add actions only after the drafts prove accurate.
The grouping below uses a FAKE scripted function in place of a model. Real signatures come from log text, not single letters.
import sys
def group_by_signature(logs):
groups = {}
for sig in logs:
groups[sig] = groups.get(sig, 0) + 1
return groups
def main():
logs = ["A"] * 5 + ["B"] * 4 + ["C"] * 3
groups = group_by_signature(logs)
total = 0
for sig in groups:
total = total + groups[sig]
if total != 12:
print("FAIL: grouped total is off")
sys.exit(1)
if len(groups) != 3:
print("FAIL: signature count is off")
sys.exit(1)
if sorted(groups.values()) != [3, 4, 5]:
print("FAIL: group sizes are off")
sys.exit(1)
agent_min = 12 * 3
review_min = len(groups) * 5
assisted = agent_min + review_min
saved = 240 - assisted
if agent_min != 36:
print("FAIL: agent minutes are off")
sys.exit(1)
if review_min != 15:
print("FAIL: review minutes are off")
sys.exit(1)
if assisted != 51:
print("FAIL: assisted total is off")
sys.exit(1)
if saved != 189:
print("FAIL: saved minutes are off")
sys.exit(1)
p = 900
chain_th = p * p * p // 1000000
if chain_th != 729:
print("FAIL: chain thousandths off")
sys.exit(1)
print("PASS: 12 failures group into 3 signatures with 51 assisted minutes and 189 minutes saved")
main()schematic draft (illustrative example, not real model output):
Group A (5 failures): reset deassertion mismatch in uart_tb, first at 120 ns.
Group B (4 failures): fifo full flag late by one cycle in fifo_tb.
Group C (3 failures): timeout waiting for grant in arb_tb.
Suggested next step for each group: point at the quoted log lines for human review.Manual work is 240 minutes and assisted work is 51 minutes. The saving is 189 minutes, or 3.15 h.
0.9 x 0.9 x 0.9 = 0.729 exactly.