Fluency means the text reads smoothly and follows likely patterns. Correctness means each claim matches evidence, tests, or sources.
The training objective rewards likely continuations, not checked facts. Smooth text can therefore state false details with the same confidence as true ones.
Signal | What it tells | What it never tells |
|---|---|---|
Smooth grammar | The text follows common patterns | Nothing about factual truth |
Confident tone | The continuation scored highly | Nothing about evidence |
Specific details | The pattern includes concrete names | Nothing about whether they exist |
chain success for 5 independent claims, exact
assume each claim is correct with probability 0.9 (stated EXAMPLE value)
p(all 5 correct) = 0.9^5 = 0.9^2 x 0.9^2 x 0.9 = 0.81 x 0.81 x 0.9
= 0.6561 x 0.9 = 0.59049
result: 0.59049, dimensionless probabilityTraining uses a collection gathered up to some date. Events after that date are absent, so answers about them are reconstructions rather than records.
The exact cutoff varies by model and deployment, so confirm it for the system in use. Anything dated after the confirmed cutoff needs an external source.
schematic illustration, not real model output, no specific system implied
prompt: describe the release from last month
fluent reply: a confident paragraph with version numbers and dates
problem: training collection ended before that month
verdict: every dated claim needs an external sourceTreat every substantive claim as unverified until checked. Tests, simulation, datasheets, and source documents decide, not tone.
Verify code by running it and reviewing the diff before accepting it.
Verify facts against current sources when dates or versions matter.
Keep credentials and secrets out of prompts; share proprietary material only through an account and deployment approved for its data class and task; use least-privilege excerpts.
Treat files and logs as untrusted input that can carry injected instructions.
Require human approval before risky actions such as commits, tapeout changes, or releases.
A frequent mistake is quoting fluent detail into a spec or log summary without a check. Detail density is a style property, not evidence.
0.59049. Five independent 0.9 factors multiply to 0.59049, so long fluent answers compound small per-claim risks.
Treat every dated claim as unverified. Check a current external source before using any of it.