A memory controller reads a cache line and finds an uncorrectable ECC error in it. What should it do? If it drops the read, the requester waits for ever (lesson 5.2). If it raises a fatal error, the whole server crashes, perhaps for a line that belongs to a process that will never read it again. If it returns the data as normal, corrupted data flows silently into computation. There is a fourth option.
What should a system do with data it knows is bad?
The obvious choices are all poor. Dropping the transaction breaks the one-response rule. Returning an error response stops this requester, but the data may be on its way somewhere else (a cache fill, a DMA transfer) where nobody checks the status. Crashing the system is safe but punishes everyone for one bad line that may never be used. What is needed is to let the transaction complete normally while making sure nobody can use the bad data without noticing.
Poison is a marker attached to the data itself: "this data is known to be bad". The transaction completes, the data moves on through caches, buffers and links, and the marker moves with it. Nothing fails until something actually consumes the data (a processor loads it into a register, a device acts on it). At that point the consumer raises an error, precisely, for that use.
Poison is one part of containment: making sure an error affects as little as possible. The principles:
Mark, do not spread silently: corrupted data must never lose its marker (a bridge that strips poison turns a contained error into silent corruption).
Fail the user, not the system: the process that consumes bad data gets an error; others continue.
Stop writes of bad data reaching storage unmarked: a write carrying poison should keep it in memory (or be refused), so a later reader also sees the problem.
Escalate when containment is impossible: if poison cannot be carried (a protocol without a poison bit), the safe choice is an error response or a fatal error, not clean-looking data.
Protocol | Poison |
|---|---|
PCI Express | A poisoned TLP has its EP bit set; it is delivered to the receiver's application layer, which decides what to do; it is reported as a non-fatal error. |
Coherent interconnects and processors | Poison bits on data in caches and fabrics, raising a precise error on consumption. |
Protocols without poison (simple buses) | Must use an error response instead, and lose the data. |
A server design (illustrative) crashed the whole machine on every uncorrectable memory error. Most of those errors were in pages that were never read again, or belonged to one process. Adding poison let the memory controller mark the line, let the processor raise an error only when a process loaded it, and let the operating system end just that process. Machine-wide crashes from memory errors dropped to almost none.
Nothing visible: it is eventually evicted or overwritten; no error is raised because nobody consumed it.
Stripping turns known-bad data into clean-looking data, which flows on silently; an error response at least tells the requester.
Poison marks known-bad data so the transaction completes but the data cannot be used unnoticed.
The error is raised only where the data is consumed, containing it to that user.
Every cache, bridge and link must carry the marker; where it cannot, escalate instead.