You do not. You make it fail every run first. A coherence race fails only when two transactions overlap in a narrow window, and random traffic hits that window rarely. Reproduction is a sequence of steps that each raise the hit rate, until the failure is deterministic and small enough to read in a trace.
Step | What you change | Why it helps |
|---|---|---|
1. Narrow the address set | Run the failing workload on one line or a few lines. | Every transaction now conflicts with every other; the overlap rate rises by roughly the reduction in addresses. |
2. Inject delay at the suspected class | Hold one message class (writebacks, Inv-Acks, forwards) for many cycles. | Widens the race window from a few cycles to most of the run. The failure is real, valid when the delay respects every ordering the protocol specification requires. |
3. Reproduce off silicon | Run the same test on an emulator or FPGA, or in the reference model. | Full visibility, and the delay setting can be swept. |
4. Shrink | Remove operations while the failure still reproduces. | A test of three operations can be read in a trace and becomes a regression test. |
5. Confirm the fix | Run the shrunk test and the original workload on the fixed design, with the same delays. | The shrunk test must pass; the original must stop failing. |
Step 3 uses the reference model and injected-bug tests built in Coherency Scoreboards and Reference Models: a model that runs the same operation list can tell you whether the failure is in the protocol or in the implementation. Determinism and shrinking in general are covered in Shrinking a Failing Test.
Illustrative case. A stress test reports, roughly once per thousand runs, a load that returns an older value of X. The suspicion falls on Eviction Versus Intervention, Order B: C0 evicts X with PutM W, and C1's GetS R wins at H while W is still in flight. In the correct design C0 is in MI^A and answers the Fwd-GetS from its writeback entry. The suspected bug: C0's forward lookup checks only the cache array, finds no tag and answers 'no data', so H reads memory, which is stale, and gives C1 the old value. The late PutM then arrives from a non-holder and gets Put-Ack with no change, so the new value is lost.
The race window is the time PutM spends in flight, a few cycles. The check below models it: random tests of 30 loads, stores and evictions from two cores, then the same tests on one address, then one address with PutM delayed by 50 time units. It then shrinks a failing test by removing one operation at a time. A local access to a line the core has in MI^A stalls until Put-Ack, as in the course model.
import random
def run(test, buggy, inject):
"""Replay ops (gap, core, kind, addr). Return True if a load saw a stale value."""
mem, holder, val, golden, wb, t = {}, {}, {}, {}, {}, 0
def deliver(now):
for a in [a for a, (c, v, due) in wb.items() if due <= now]:
c, v, _ = wb.pop(a)
if holder.get(a) == c: # H still records C as holder: write memory
mem[a], holder[a] = v, None
# else: PutM from a non-holder, Put-Ack only, no change
def take(c, a): # obtain the newest value for core c
h = holder.get(a)
if h is None or h == c:
return val[a] if h == c else mem.get(a, 0)
if a in wb and wb[a][0] == h: # holder is in MI^A, data sits in its WB entry
v = mem.get(a, 0) if buggy else wb[a][1] # bug: Fwd misses the WB entry
else:
v = val[a]
mem[a], holder[a] = v, None # dirty data reaches memory via H
return v
for gap, c, kind, a in test:
t += gap
if a in wb and wb[a][0] == c: # own line in MI^A: stall until Put-Ack
t = max(t, wb[a][2])
deliver(t)
if kind == "ld":
if take(c, a) != golden.get(a, 0):
return True
elif kind == "st":
take(c, a)
golden[a] = golden.get(a, 0) + 1
holder[a], val[a] = c, golden[a]
elif kind == "ev" and holder.get(a) == c and a not in wb:
wb[a] = (c, val[a], t + (50 if inject else 2)) # PutM in flight, MI^A
return False
def random_test(rng, n_addr, n_ops=30):
return [(rng.randint(1, 20), rng.randint(0, 1), rng.choice(["ld", "st", "st", "ev"]),
rng.randrange(n_addr)) for _ in range(n_ops)]
def fail_rate(n_addr, inject, buggy=True, trials=1000):
rng = random.Random(7)
return sum(run(random_test(rng, n_addr), buggy, inject) for _ in range(trials)) / trials
base = fail_rate(8, inject=False)
narrow = fail_rate(1, inject=False)
forced = fail_rate(1, inject=True)
assert base < 0.05, base # rare: the bug escapes most random tests
assert base < narrow < forced and forced > 0.25, (base, narrow, forced)
for n_addr in (1, 8): # the fixed controller never fails
for inj in (False, True):
assert fail_rate(n_addr, inj, buggy=False) == 0
def shrink(test):
"""Greedy one-op removal while the failure still reproduces."""
changed = True
while changed:
changed = False
for i in range(len(test)):
cand = test[:i] + test[i + 1:]
if run(cand, True, True):
test, changed = cand, True
break
return test
rng = random.Random(7)
failing = next(tc for tc in (random_test(rng, 1) for _ in range(100)) if run(tc, True, True))
small = shrink(failing)
kinds = [(c, k) for _, c, k, _ in small]
assert len(small) == 3 and kinds[1][1] == "ev" and kinds[0] == (kinds[1][0], "st"), small
assert kinds[2] == (1 - kinds[1][0], "ld"), small
assert not run(small, False, True) # the same shrunk test passes on the fix
print(f"PASS: base {base:.3f}, one line {narrow:.3f}, one line + PutM delay {forced:.3f}, shrunk to {len(small)} ops")With eight addresses and normal timing the buggy design fails about 1 test in 1,000: the escape rate that made the bug rare. On one address it fails about 16 in 1,000, and with PutM delayed about 300 in 1,000. The fixed design never fails under any setting, so the delay did not create the failure. Shrinking leaves three operations: one core stores X, the same core evicts X, the other core loads X while the PutM is in flight. In the shrunk test the evicting core happens to be C1; renamed to the linked lesson's roles it is C0 stores, C0 evicts, C1 loads, which is exactly Order B, and the shrunk test passes on the fixed model.
If no single class raises the rate, the window may need two delays (one class held and another sped up by delaying everything else), or the suspect mechanism is wrong. A delay sweep that never raises the rate is evidence too: it says the failure is probably not a race on that class. The stale-data triage lesson helps choose the first suspect from the symptom.
It shows the delay does not cause failures on its own: it only exposes a bug that the design already has. If the fixed model also failed under the delay, either the fix is incomplete or the injected behaviour is not one the protocol allows, and the reproduction would prove nothing about silicon.