Assistant output reads fluently even when it is wrong. This topic catalogues the failure modes that matter on engineering tasks, with a detection check and an engineering example for each.
Mode | Typical sign | Detection check |
|---|---|---|
Invented names | A flag, call or signal that sounds right | Look up every name in the real reference. |
Wrong numbers | Confident arithmetic or a stale fact | Recompute from stated inputs and current tables. |
Tone and agreement | Certainty, or agreement with a wrong premise | Test the claim against counter-evidence. |
Instruction drift | Late turns drop early constraints | Rescan the session against the stated contract. |
Bugs that compile | Plausible code with a hidden defect | Run boundary tests that cover the range ends. |
Lesson | What it covers |
|---|---|
Fluent but invented names, with reference checks and a flag audit program. | |
Wrong arithmetic and stale facts, with recomputation and repricing checks. | |
Certainty as style and agreement with wrong premises, with counter-checks. | |
Fading constraints over many turns, with restatement and rescan habits. | |
Plausible code with hidden defects, caught by boundary testbenches. |
Treat the name as invented until the reference confirms it. Fluency is not evidence that the flag exists.