A team builds a beautiful random testbench for an image scaler: millions of frames, every resolution, every mode. Then someone asks how the testbench knows each output frame is correct. The answer is that it does not: it checks the frame has the right size and no X values. The stimulus was sophisticated; the judgement of right and wrong was almost absent. Without an oracle, a testbench is a very expensive way to watch a design run.
What decides whether an output is correct, and why is that often the hardest part to build?
The obvious answer is "the specification decides". But a specification is prose, and prose cannot compare two numbers. Something executable has to turn the specification into an expected answer for every input the stimulus produces, including inputs nobody wrote down. That something, the oracle, must be correct, complete enough for the stimulus, and independent of the design. Building it can take as much effort as the design itself.
Oracle | How it knows the answer | Good for | Weakness |
|---|---|---|---|
Rules (assertions) | Checks properties every output must satisfy | Protocols, invariants, local behaviour | Says what is forbidden, not the exact answer |
Reference model | Computes the expected output from the input | Data transformations, routing, arithmetic | A second implementation, with its own bugs |
Golden results | Stored outputs from a trusted source | Fixed tests, regression of known cases | Only for inputs someone ran before |
Inverse or round trip | Undo the operation and compare with the input | Encoders and decoders, compressors | Misses errors the inverse also makes |
Another implementation | Compare with a previous version or a different design | Re-implementations, optimisations | Shared mistakes look like agreement |
People | An expert inspects the result | Images, audio, early bring-up | Slow, not repeatable |
Correctness: an oracle with a bug either hides design bugs or reports false ones.
Coverage of the stimulus: it must give an answer for every input the stimulus can produce, not just the cases someone wrote down.
Independence: written from the specification by someone other than the designer (lesson 0.3).
The right precision: predict what the specification promises, and no more (lesson 2.7).
Practice | Purpose |
|---|---|
Reference models written in a high-level language | Easier to write and review than RTL; a different implementation |
Assertions for protocol and invariant rules | Cheap, local oracles that need no expected values |
Golden data from algorithm teams | An independent source for numerical results |
Oracle reviews against the specification | The oracle is code, and its bugs are verification bugs |
The opening scaler (illustrative) ran for months with a size-and-X check as its only oracle. When an algorithm team's Python reference was finally connected, the first night found eleven bugs, including a rounding error visible in every frame. The stimulus had exercised all of them from the start; nothing had been able to say the output was wrong.
Only bugs that produce unknown values: wrong data, wrong order or wrong timing all pass, because nothing knows the right answer.
Mistakes shared by both versions look like agreement, and intended changes look like mismatches that must be explained away.
Every check needs an oracle: something that knows the right answer for each input.
Oracles include rules, reference models, golden data, inverses, other implementations and people, each with a weakness.
A testbench is only as good as its oracle; budget, review and validate it like design code.