First test the checker, then test the assistant. The listing below supplies fixed reply fixtures to small property checkers. It is a grader unit test: it does not send a prompt to a model and cannot measure whether changing a prompt improves an answer. Keep this inexpensive check separate from the prompt evaluation described after the listing.
Part | Role | In the program |
|---|---|---|
Fake model | Returns fixed replies per case id. |
|
Cases | Pairs each id with two property checks. |
|
Checkers | Test one property of a reply string. | Six small functions. |
Runner | Counts passing checks and prints one line. |
|
The program below is runnable and uses only the standard library. Its fake model is scripted, with no real model calls.
import json
import sys
# FAKE model stand-in: scripted replies per case id, no real model calls.
def fake_model(case_id):
table = {
"reset-steps": "STEP 1: hold reset_n low for 4 cycles\nSTEP 2: release reset_n\nSTEP 3: wait 2 cycles",
"width-note": "The 8-bit counter wraps at 255 and raises full.",
"config-shape": '{"width": 8, "reset_cycles": 4}',
"secret-handling": "CREDENTIAL: redacted. STATUS: ok.",
"product-steps": "20 * 35 = 700\n4 * 35 = 140\nTOTAL = 840",
"bullets": "- hold reset_n low for 4 cycles\n- release reset_n and wait 2 cycles",
}
return table[case_id]
def has_text(reply, text):
return text in reply
def starts_with(reply, text):
return reply.startswith(text)
def line_count(reply, n):
return sum(1 for ln in reply.splitlines() if ln.strip()) == n
def excludes(reply, text):
return text not in reply
def json_keys(reply, keys):
try:
obj = json.loads(reply)
except ValueError:
return False
return isinstance(obj, dict) and sorted(obj.keys()) == sorted(keys)
def bullet_lines(reply):
lines = [ln for ln in reply.splitlines() if ln.strip()]
return len(lines) > 0 and all(ln.startswith("- ") for ln in lines)
CASES = [
("reset-steps", [("starts-with", "STEP 1:"), ("line-count", 3)]),
("width-note", [("contains", "8-bit"), ("excludes", "TODO")]),
("config-shape", [("json-keys", ["width", "reset_cycles"]), ("contains", "4")]),
("secret-handling", [("excludes", "ACME-SECRET-99"), ("contains", "redacted")]),
("product-steps", [("contains", "TOTAL = 840"), ("line-count", 3)]),
("bullets", [("bullet-lines", None), ("contains", "cycles")]),
]
def run_one(check, reply):
kind, arg = check
if kind == "contains":
return has_text(reply, arg)
if kind == "starts-with":
return starts_with(reply, arg)
if kind == "line-count":
return line_count(reply, arg)
if kind == "excludes":
return excludes(reply, arg)
if kind == "json-keys":
return json_keys(reply, arg)
if kind == "bullet-lines":
return bullet_lines(reply)
return False
def main():
passed = 0
total = 0
for case_id, checks in CASES:
reply = fake_model(case_id)
for check in checks:
total += 1
if run_one(check, reply):
passed += 1
if passed == total:
print("PASS: %d of %d property checks hold across %d cases" % (passed, total, len(CASES)))
else:
print("FAIL: %d of %d checks hold across %d cases" % (passed, total, len(CASES)))
sys.exit(1)
main()A green run means the supplied fixtures satisfy these checkers, not that a prompt works. A broken checker can still accept wrong answers: the width-note fixture mentions an 8-bit counter but that alone does not establish correct wrap behaviour. Add deliberately incorrect fixtures and require their rejection before trusting a grader.
harness size follows the code: 6 cases x 2 checks = 12 checks
green run: 12 / 12 = 100%
a red fixture run means inspect the fixture and checker; prompt changes are judged on actual outputsPrepare a separate evaluation record with case_id, prompt_version, full input, model/setup, output, checks, usage and outcome. Obtain outputs through a company-approved assistant or an approved model adapter; do not place credentials in the record. An offline learner can inspect clearly labelled recorded outputs, but must not describe those as a run of their own prompt. Live evaluation is useful precisely because model behaviour varies.
Step | What to do | Evidence to keep |
|---|---|---|
Freeze | Select public or synthetic cases and write expected properties before looking at replies. | Inputs, requirements and known traps. |
Run | Give the same unseen inputs to prompt A and prompt B. Repeat where output variability matters. | Actual replies and setup for every attempt, including failures. |
Check | Use the tested graders, then inspect technical meaning and source support. | Case-level correctness, omissions and human review. |
Decide | Compare quality and effort without tuning on the held-out cases. | What improved, what regressed and what remains uncertain. |
Not on this evidence. This harness has no prompt input. Inspect the fixture and checker, then separately evaluate the prompt against actual outputs.
Fixtures make checker defects reproducible. Actual model trials measure task performance and variability. Both are useful, but neither substitutes for the other.