Question 1: the assistant hands you a script

Tests whether the candidate runs and reads before merging. Strong answers treat the draft as unreviewed code from a stranger.

Read the diff, run the tests, and check the boundary cases yourself. A second independent method beats a second glance at the same output.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

def fake_assistant_total(lines, plant_bump):
    # FAKE scripted stand-in for a model, not real output
    return sum(lines) + plant_bump


def verify_total(lines, claimed):
    return sum(lines) == claimed


bad = []
lines = [412, 87, 231]
honest = fake_assistant_total(lines, 0)
slipped = fake_assistant_total(lines, 50)
if honest != 730:
    bad.append("honest total should be 730")
if not verify_total(lines, honest):
    bad.append("honest total should verify")
if verify_total(lines, slipped):
    bad.append("bumped total should not verify")
if bad:
    print("FAIL: " + "; ".join(bad))
    raise SystemExit(1)
print("PASS: recompute accepts 730 and rejects the bumped total")

The totals above are scripted fakes, not model output. The program proves that independent recomputation catches the planted 50-count bump.

Question 2: the claim cannot be run

Tests honesty when no test exists. Strong answers bound the claim and name what would change their mind.

Cross-check with a second source and state the uncertainty plainly. Two checks that read the same output are one check, not two.

1
2
3
4

EXAMPLE parameters: each independent check misses a defect with chance 0.1
both miss = 0.1 x 0.1 = 0.01
caught by at least one = 1 - 0.01 = 0.99
checks must be independent: re-reading the same output twice is one check, not two

Question 3: secrets and untrusted content

Tests practical safety with proprietary code. Strong answers name least privilege, redaction, and approval.

Keep credentials and secrets out of prompts; share proprietary material only through an account and deployment approved for its data class and task. Treat files and logs as untrusted: they can carry instructions that the model may follow without noticing.

Common mistakes

Related topics

• Prompting questions

• Verifying AI output

• Failure modes

• Data, privacy and IP