A post-silicon coherence regression is a fixed set of tests with fixed seeds and known pass criteria, run on real parts. Its job is to find coherence and ordering bugs, and then to prove that later changes (firmware, microcode, configuration, chicken bits) have not reopened them. No test in it needs a reference model at run time: each one checks itself.
Component | Example contents | Pass criterion |
|---|---|---|
Litmus suites | SB, MP, IRIW, CoRR, their fenced variants, and generated families over many thread placements | No forbidden outcome, for the named model |
Hot-line stress | All cores reading and writing one line, with flushes in between | Self-checking values, no hang |
False sharing | Each core writes its own word in one shared 64-byte line | Every core reads back exactly its own writes |
Lock storms | All cores take one lock and increment a shared counter N times each | Counter = cores x N |
Eviction pressure | Background threads sweep conflicting sets during other tests | The other tests still pass |
Random multi-threaded tests | Seeded random loads, stores and atomics with value signatures | The signature checks below |
Power transitions under traffic | A core leaves and rejoins the coherence domain while others access its lines | No lost dirty data, no hang |
Configuration matrix | The suites above with each shipped chicken bit set | Same verdicts as the default |
False sharing and hot lines come from Part 7: here they are used as stress, not as performance problems. Power transitions are taught in Part 10, in Leaving and Rejoining the Domain.
Silicon gives no log of every transaction, so the test data must carry the evidence. Each core writes only its own location, with values that encode (writer, sequence number), always increasing. A reader can then check three things locally: every value it read was written by someone; its own reads of one location never go backwards (CoRR); and lock-protected counters end at the exact total. The model below runs a correct memory, one with a rare injected stale read, one whose lock is sometimes not exclusive (each increment is a read then a write of the counter, so two cores can read the same old value), and one read of a value nobody wrote.
import random
def run(seed, stale_rate=0.0, lock_ok=True, bogus_at=None, cores=3, steps=3000):
"""Each core owns one location and writes (core, seq) with seq rising.
Every core also reads random locations and does locked increments of CNT,
each one a read of CNT then a write of CNT + 1.
stale_rate > 0: a read sometimes returns an older copy (injected bug).
lock_ok=False: the lock is sometimes not exclusive, so a second core
reads the same old CNT before the first core writes it back.
bogus_at: index of one read that returns a value nobody wrote."""
rng = random.Random(seed)
mem = {c: (c, 0) for c in range(cores)}
history = {c: [(c, 0)] for c in range(cores)}
cnt, increments = 0, 0
reads = [] # (reader, location, value)
for _ in range(steps):
core = rng.randrange(cores)
action = rng.random()
if action < 0.3: # write own location
seq = mem[core][1] + 1
mem[core] = (core, seq)
history[core].append(mem[core])
elif action < 0.8: # read any location
loc = rng.randrange(cores)
value = mem[loc]
if rng.random() < stale_rate and len(history[loc]) > 2:
value = history[loc][-3] # the injected stale copy
if len(reads) == bogus_at:
value = ((loc + 1) % cores, 1) # never written here
reads.append((core, loc, value))
else: # locked increment
r = cnt # read CNT under the lock
if not lock_ok and rng.random() < 0.01:
r2 = cnt # second core inside the lock
cnt = r2 + 1 # it writes CNT
increments += 1
cnt = r + 1 # first core writes CNT
increments += 1
return reads, mem, cnt, increments
def check(reads, mem, cnt, increments):
"""Self-checking signature: needs no reference model."""
last = {}
for reader, loc, (writer, seq) in reads:
if writer != loc or seq > mem[loc][1]:
return "value never written"
if seq < last.get((reader, loc), 0):
return "read went backwards" # CoRR violation
last[(reader, loc)] = seq
if cnt != increments:
return "lost increment"
return "ok"
for seed in range(20):
assert check(*run(seed)) == "ok" # correct design passes every seed
caught = [check(*run(seed, stale_rate=0.001)) for seed in range(20)]
assert caught.count("read went backwards") >= 15 # rare stale reads are caught
lost = [check(*run(seed, lock_ok=False)) for seed in range(20)]
assert lost.count("lost increment") >= 15 # a non-exclusive lock is caught
assert check(*run(3, bogus_at=50)) == "value never written"
print("PASS: signature checks pass a coherent run and catch stale reads, lost increments and invented values")The checks have a limit worth knowing. A stale read is caught only if the same reader had already seen a newer value of that location; a stale first read looks legal. That is why such tests read each location many times, and why they are combined with flag-and-payload tests (MP with fences), where reading the flag proves which payload value must follow.
A large regression produces many failures from few bugs. Group them into buckets by a signature, not by the symptom text. A useful signature records the failing check (which test, which assertion), the address class (same home, cross-die, a region), the configuration (which chicken bits were set), and the reproduction rate at the stored seed and placement. Two failures that both say "wrong value" but disappear under different chicken bits are two bugs. The triage topic of this Part covers what to do with one bucket: classify against the model, then trace and localise.
Once the regression exists, it gates every change that can alter coherence behaviour: new firmware, microcode, power-management code, configuration defaults and chicken-bit settings. Keep the baseline outcome histograms per test and placement; a histogram whose allowed outcomes suddenly stop appearing means the stress has weakened, which is also a regression. A forbidden outcome is never waived as flaky: it blocks the change until it has a bucket and an owner.
Atomicity of the lock-protected update: two increments were lost, so two cores held the lock or saw the line as writable at once (an SWMR or atomic read-modify-write failure). Other suspects give the same count: a lost writeback of the counter line, a reservation (load-reserved or load-exclusive monitor) not cleared when another core wrote the line, or a missing acquire/release on a weak model, so the counter update leaks out of the critical section. Triage separates them. The value check found the failure without any trace of which transaction failed.