A processor runs a loop that copies a buffer: read a word, write it somewhere else, repeat. Speeding up the memory's write path makes almost no difference to the loop. Speeding up the read path makes it nearly twice as fast. Reads and writes both move one word across the same bus, so why does one matter so much more?

Why are reads and writes not symmetric, and what does that mean for protocol design?

The obvious answer, and why it breaks

The obvious picture is that a read and a write are the same transaction pointed in opposite directions: an address goes out, a word moves, done. But look at what the requester needs before it can carry on. After a write, it needs nothing back: the data has left, and something downstream can finish the job. After a read, it needs the data itself, and that cannot arrive until the request has reached the target and the answer has come all the way back. A read always costs a full round trip; a write need not cost any.

The idea: data flows differently, so "done" means different things

Worked numbers

A copy loop with illustrative numbers: memory 100 ns away, and a write buffer in the interconnect that accepts a write in 2 ns.

Per word copied:
  read:  request out + data back                    = 100 ns  (the loop must wait for it)
  write: handed to the write buffer                 =   2 ns  (memory finishes it later)
  loop time per word                                = 102 ns
Make writes twice as fast (1 ns):   101 ns per word  (1% faster)
Make reads  twice as fast (50 ns):   52 ns per word  (nearly 2x faster)

The write path matters only for bandwidth: the buffer must drain as fast as writes arrive, or it fills and writes start to wait too. The read path matters for latency, and latency is what the loop sees.

Consequences for protocol design

• Separate read and write paths. Reads and writes carry data in opposite directions, so a protocol that gives each its own channels can move read data and write data at the same time (lesson 5.6).

• Many reads in flight. Since a read's latency cannot be hidden, protocols let a requester issue many reads before the first answer returns (Part 6).

• Writes may not wait for an answer. A posted write gets no response at all, and its requester never waits (lesson 5.7). The price is knowing less about when, or whether, the write finished.

• Reads are where ordering bites. A read can be answered from a buffer only if no newer write to the same address is still waiting in another buffer (lesson 7.8).

How real protocols treat them

What breaks

A design team (illustrative) sized a bridge's buffers with equal space for reads and writes, reasoning that traffic was half each. Writes flowed through at full rate because the bridge accepted them at once. Reads queued behind the slow target, each holding a requester idle, and the requesters starved. The writes needed buffer space for bandwidth; the reads needed more outstanding slots to cover their latency, which a symmetric design never gave them.

Key takeaways

• A read must wait a full round trip for its data; a write carries its data and can be handed off at once.

• Read latency stalls the requester; write performance is about bandwidth and buffering.

• Protocols separate read and write paths, allow many reads in flight, and often post writes.

Related topics

• Part 5 overview: Transactions