Question 1: how do you get output a script can parse

Tests whether the candidate constrains the format instead of hoping. Strong answers name the format, show one example, and validate.

Ask for named fields with one exact example line. Then parse strictly and reject anything else, since no technique works on every model.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

def fake_assistant(prompt):
    # FAKE scripted stand-in for a model, not real output
    if "FORMAT: id, verdict" in prompt and "log line" in prompt:
        return "id: 42, verdict: timeout"
    return "it timed out maybe"


def parses_as_record(reply):
    parts = [p.strip() for p in reply.split(",")]
    if len(parts) != 2:
        return False
    return parts[0].startswith("id: ") and parts[1].startswith("verdict: ")


bad = []
structured = fake_assistant("FORMAT: id, verdict. log line 42 timed out")
plain = fake_assistant("what happened in the log")
if not parses_as_record(structured):
    bad.append("structured prompt should parse")
if parses_as_record(plain):
    bad.append("plain prompt should not parse")
if bad:
    print("FAIL: " + "; ".join(bad))
    raise SystemExit(1)
print("PASS: constrained prompt parses, unconstrained prompt does not")

The stand-in above is fake and scripted, never real output. It proves the point that the constraint decides parseability, not the wording around it.

Question 2: when do examples and reasoning steps help

Tests measured judgment about few-shot prompting. Strong answers say it varies by model and propose a comparison.

Examples help when the format is unusual or the boundary cases matter. What works differs between models, so measure with and without on a fixed set.

1
2
3
4

schematic transcript (illustrative, not real model output)
Q: decide pass or fail from: id 42 ran 512 cycles, timeout at 500 cycles
A: id 42 fails: 512 > 500, over by 512 - 500 = 12 cycles
rule shown: quote the numbers, then the comparison, then the verdict

The transcript is schematic and invented for teaching. It shows the shape of a checkable answer: numbers first, verdict last.

Question 3: the prompt works sometimes

Tests debugging discipline for flaky prompts. Strong answers isolate one change at a time and keep a log.

Temperature rescales the sampling distribution, and temperature 0 is approximately greedy. That is not a guarantee of identical output across runs or hardware, so log prompts and outputs to tell flakiness from drift.

Common mistakes

Related topics

• Verification questions

• Anatomy of a prompt

• Examples and reasoning

• Prompt debugging