A device asks for 512 bytes in one read request. The data comes back as three packets: 208 bytes, 256 bytes, 48 bytes. The device's designer had assumed one request means one response, freed the request's tag on the first packet, and handed it to a new read. The other two packets were then delivered to the new read.
Why would one request be answered in several pieces, and what rules govern the pieces?
The obvious model is one request, one response. But a request can ask for more data than one response packet may carry: the largest packet a link accepts is limited so that buffers stay small and a big packet does not hold the link for too long. A completer may also want to start sending the first part of a large read before it has fetched the rest. So large reads come back in pieces, and the requester must expect it.
A request may ask for up to a maximum read request size; a completion may carry at most a maximum payload size. When the first exceeds the second, the answer must be split.
Splits may happen only at naturally aligned completion boundaries (for example every 64 or 128 bytes), so every piece starts or ends on a predictable address and receivers can place data in aligned chunks.
The pieces of one request come back in address order, each carrying the same tag and a count of bytes still to come.
The request's tag stays reserved until the last piece arrives (lesson 6.4).
Maximum payload 256 bytes, completion boundary 64 bytes, a 512-byte read starting at 0x1030 (illustrative settings):
Read covers 0x1030 to 0x122F = 512 bytes
Piece 1: 0x1030 up to the boundary 0x1100 = 208 bytes (ends on a 64-byte boundary, <= 256)
Piece 2: 0x1100 to 0x11FF = 256 bytes (the maximum payload)
Piece 3: 0x1200 to 0x122F = 48 bytes (the end of the request)
Total = 512 bytesOther splits would also be legal (for example 16 bytes to 0x1040, then aligned pieces), so a requester must handle any legal split, not one pattern.
Duty | Why |
|---|---|
Keep the tag until the byte count reaches zero | Freeing it on the first piece hands later pieces to the wrong request |
Have buffer space for the whole request when it is issued | Pieces may arrive back to back; refusing one blocks completions for other requests behind it |
Place each piece by its address | Pieces have different sizes depending on alignment |
Plan for the maximum number of pieces | Some designs track completions per request; size those tables for the worst split |
Protocol | Split completions |
|---|---|
PCI Express | A read larger than the maximum payload size (128 to 4096 bytes, configured per link) returns as several completions, split only at the Read Completion Boundary (64 or 128 bytes for a root complex, 128 bytes for other components) and returned in increasing address order. |
AXI | One read burst returns its beats on the read-data channel with RLAST on the last; the equivalent of splitting is a long burst arriving as many beats, possibly interleaved with others (lesson 6.7). |
The opening bug (illustrative), and its cousin: a device tested against one completer that always returned whole 256-byte pieces was connected to a root complex that split at 64-byte boundaries. Its completion tracker had room for at most 2 pieces per read and overflowed on the third. Test against completers that split as finely as the rules allow.
0x2010–0x207F (112 bytes, ending on a 128-byte boundary), 0x2080–0x210F (144 bytes, the end of the request). It could also come back whole, 256 bytes, if the completer chooses.
Each completion carries a count of bytes remaining (or the total and a position); the read is complete when the remaining count reaches zero.
A read larger than the maximum payload returns as several completions, split only at aligned boundaries and in address order.
The requester keeps the tag and the buffer space until the last piece arrives.
Design and test for the worst legal split, not the usual one.