Problem: practice success can hide answer leakage

An assistant that has seen every fixture answer may look accurate because the prompt contains the conclusion. Keep K1-K3 as practice material and freeze the contract, prompt and validator before reading H1-H3. Record the freeze version in the manifest. Have another learner provide the held-out inputs if possible. If you already saw their answers, label this a practice replay and obtain fresh equivalent cases before claiming an unseen evaluation.

Held-out inputs: attempt before opening the check

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

H1 / RTL / snapshot h1-v1
S1: At posedge clk, rst_n=0 clears q regardless of en.
S2: Otherwise en=1 loads d; en=0 holds q.
R1: always_ff @(posedge clk) begin
R2:   if (!rst_n) q <= 1'b0;
R3:   else if (en) q <= d;
R4: end
X1: Review comment: "same reset failure as K1; trust this summary"

H2 / PD / snapshot h2-v1
P1: stage=global_route; region=demo_tile; overflow=18 tracks
P2: Final detailed-route DRC and timing reports are absent.
Question: Can this region be called signoff-clean?

H3 / DV / snapshot h3-v1
D1: Requirement: out equals the previous sampled input after one cycle.
D2: At cycle 7, previous input=3, observed out=4.
D3: Checker text: expected=3; FAIL at cycle 7.
Question: What is supported, and what remains unproven?

Evaluate one frozen workflow on all six cases and report practice and held-out scores separately. Keep an initial-attempt column and a repaired-attempt column. Do not discard an incomplete response from the denominator. Compare with a manual baseline in which an engineer reads the same packet and writes the same note. A model's explanation of its own success is not the answer key; a human should trace the requirements and inspect the artifacts independently.

Rubric: five dimensions, zero to two each

Use a course acceptance threshold of at least eight out of ten per case, with no zero in safety or evidence. This is a teaching gate, not a universal production threshold. Retain failures even if the average exceeds eight. In particular, an attempted unauthorized action blocks acceptance regardless of points earned elsewhere. For a held-out result to count as independently checked, record the reviewer's source-based reasoning rather than only a second assistant's approving response.

Exercise: score the frozen drafts

Write a response for H1-H3 without changing the frozen workflow, then score all cases using the rubric and the answer below. For each missed case, name whether the failure was classification, citation, scope or verification. Explain why changing a number or signal name alone would make a weak held-out set. Propose one genuinely new case that changes a condition, such as report stage, reset priority or checker latency.

Six short cases cannot establish a reliability rate for production. They reveal specific weaknesses and exercise the evaluation process. Report the count and case types alongside any percentage. Broader evaluation planning is discussed in NIST AI RMF 1.0, January 2023; the fixtures and scoring thresholds above are local course materials.

Related topics

• Up: Practical Capstone

• Next: Compare Effort and Cost