In simulation you rerun with every signal dumped. On silicon the failure may take hours to recur, and a change in timing can hide it. So capture first: arm a debug trace buffer on the failing line, let the failure happen, freeze, and dump. Only then hypothesise. A hypothesis formed without the trace usually describes the bug you expected, not the bug you have. How trace buffers, triggers and scan dumps work in general is covered in Observability in Silicon.
A coherence trace is useful only if it can rebuild a transaction: who asked, what H decided, who answered and in what order. Each captured message needs five fields, plus two optional ones that cost record width. Leaving a field out removes a specific deduction. What to Trace sizes each field for the trace buffer.
Field | Why it is needed | What you lose without it |
|---|---|---|
Line address (mandatory) | Groups the records for one line; permissions cover the whole 64-byte line. | Records for X mix with other lines; a different word of the same line looks unrelated. |
Message type and direction or capture point (mandatory) | GetS, GetM, Inv, Inv-Ack, Fwd-GetS, Data, PutM, Put-Ack: each has a fixed role. 'in' or 'out' at H, or the tap that saw it. | You cannot tell a grant from a forwarded response, or a send from a delivery. |
Source and destination (mandatory) | Says which cache was invalidated, forwarded to or granted. | A missing Inv to C1 cannot be told from one sent to C0. |
Transaction id (mandatory) | Ties each response to the request it completes. | A late response to an old transaction looks like the answer to a new one. |
Timestamp (mandatory) | Orders the records at one capture point. H's accept (activation) order is the serialization order. It equals arrival order only if H never queues or retries; trace an accept event if it does. | You cannot place a store between two grants. |
State before and after (optional) | The cache's line state, or H's directory and pending-entry code, around the message. Cost: wires from the state arrays to the tap and a wider record. | You must rebuild every state by replaying from a known start. |
Payload word or hash (optional) | The value of the word of interest, or a hash of the line. Cost: the widest field, so fewer records fit in the buffer. | Value bugs (stale data, lost writeback) need it; permission bugs often do not. |
So: five fields plus two optional. A permission bug (a grant before an Inv-Ack) shows in the five mandatory fields alone; a value bug needs the payload option.
At the trigger, also dump the state the messages do not show: H's directory entry and pending entry for the line (requester, acks received out of required, data buffer valid), and the MSHR and line state of every cache for that line. Messages tell you what happened; the snapshot tells you what each controller believes now.
The failure report says only that C1 read 1 from X. Here is the capture from H's trace buffer, filtered on X's line. It is an illustrative generic format, not any product's output. MSI, X = 0 initially, course model.
# not-run: illustrative trace (generic format, H trace buffer, filter: line of X)
t point type src dst txn payload
1040 in GetM C0 H 0x11 -
1041 mem read H - 0x11 X=0
1046 out Data H C0 0x11 X=0, M
1212 in GetS C1 H 0x12 -
1213 out Fwd-GetS H C0 0x12 -
1239 in Data C0 H 0x12 X=1
1240 mem write H - 0x12 X=1
1247 out Data H C1 0x12 X=1, S
trigger 1250: directory S{C0,C1}, no pending entry for XRebuild it in three passes. First, sort the 'in' records by time: this H never queues or retries, so arrival order is its accept order and the serialization order (0x11, then 0x12). Second, chain records by transaction id: every response must match an open request. Third, apply the course rules to each chain. The result is the cold write and the MSI dirty read already drawn in MSI Protocol, with the same roles: C0 writes, C1 reads. The new skill is knowing what each record proves.
Record | Proves | Does not prove |
|---|---|---|
1041 mem read 0x11 | H read memory for C0's cold miss: memory held X = 0. | That no cache held the line: the directory snapshot or the earlier records must show that. |
1046 out Data 0x11 | H granted C0 M with X = 0. | That C0 received it. An 'out' record is a send, not a delivery. |
(no record) | - | C0's store of 1 is local, so H never sees it. It is inferred from the value at 1239. |
1212 in GetS 0x12 | C0's write epoch ends at this point in H's order. | When C1's load was issued at the core. |
1239 in Data 0x12 | C0 answered this forward with 1, so C0 held M / 1 until it answered: 0x11 was delivered. | Whether C0 stored more than once. |
1240 mem write | memory = 1 before C1's grant, as MSI requires before settling S/S. | - |
1247 out Data 0x12 | C1 was granted S with 1, the newest value. | Delivery, unless C1's own trace or the snapshot confirms it. |
Conclusion: C1's 1 is correct. The report was not a coherence failure, and the next question is which value software expected and under which memory model (see Ordering Bugs in Out-of-Order Cores). That is a valid outcome of trace-first debug: it can clear the coherence protocol as quickly as it can convict it.
First check the capture: overflow flag, filter settings and dropped-record counters, because a missing record may be a capture artefact. If the capture is complete, H answered from memory while C0 held the line exclusive, so it may have returned a stale value: the 'memory answered while a dirty copy existed' class of the next lesson.