When two runs disagree, the first question is whether anything about them differed on purpose. A manifest lists each run's inputs, tool versions and settings. With it, a comparison of two runs has four possible readings, and each calls for a different response.
def verdict(man_a, man_b, slack_a, slack_b, tol=1e-4):
changed = sorted(k for k in set(man_a) | set(man_b) if man_a.get(k) != man_b.get(k))
differs = any(abs(slack_a[e] - slack_b[e]) > tol for e in slack_a)
if not differs:
if changed:
return "manifests differ, results identical: check the input is actually used", changed
return "identical within tolerance", changed
if changed:
return "explained by input changes", changed
return "nondeterminism or unrecorded input", changed
base = {"netlist": "r42", "constraints": "c17", "libs": "L5", "parasitics": "p3", "settings": "s9"}
slack = {"e1": -0.020, "e2": 0.015, "e3": 0.040}
same = dict(slack)
shifted = {"e1": -0.031, "e2": 0.004, "e3": 0.040}
new_man = dict(base, parasitics="p4")
extra_man = dict(base, notes="reviewed")
r1 = verdict(base, base, slack, same)
r2 = verdict(base, new_man, slack, shifted)
r3 = verdict(base, base, slack, shifted)
r4 = verdict(base, extra_man, slack, same)
assert r1 == ("identical within tolerance", [])
assert r2 == ("explained by input changes", ["parasitics"])
assert r3 == ("nondeterminism or unrecorded input", [])
assert r4 == ("manifests differ, results identical: check the input is actually used", ["notes"])
print("PASS:", r1[0], "/", r2[0], r2[1], "/", r3[0], "/", r4[0])Manifests | Results | Verdict | Response |
|---|---|---|---|
Identical | Identical (within tolerance) | Reproducible | Nothing to do |
Differ | Differ | Explained by the changed inputs | Check the size of the shift is plausible for the change |
Differ | Identical (within tolerance) | Changed input had no effect | Check the changed input is actually used |
Identical | Differ | Nondeterminism or an input not in the manifest | Fix the flow: reproducibility is a defect |
Case 1: same inputs, same slacks: reproducible. Case 2: the parasitics version changed and slacks moved by 11 ps at two endpoints: an explained difference, but an explained verdict also needs a plausibility check that the size of the shift fits the change. Case 3: the manifests differ only in a key that should not affect timing yet slacks are unchanged: check the changed input is actually used, since a manifest entry the flow ignores gives false confidence. Case 4: the manifests are identical yet slacks differ: either the flow is nondeterministic or something that affects the result is missing from the manifest. The manifest comparison uses the union of keys, so a key present in only one manifest still counts as a change. This last case is the one that matters most, because nothing about it can be explained away, and it is also a signoff problem: you cannot reproduce your own results.
Netlist, constraint, library and parasitic versions or content hashes, per scenario.
Tool version, settings, derates, uncertainty, and any command-line options.
Host details only when the flow depends on them (thread counts, for instance).
Every reported result links to its manifest; every comparison starts by diffing manifests.
Verdicts of the third kind are treated as defects and fixed before results are used.
Manifests are generated by the flow, not typed by hand.
Explained by input changes (uncertainty), provided the size of the shift is consistent with the change in uncertainty.