A system lets writes overtake each other freely, which makes it fast. Most of the time no one cares. But at the end of filling a buffer, the producer must be sure every data write has landed before it writes the flag. Keeping everything in order all the time would give up the speed. What the producer needs is a way to say, at that one point: "finish everything before this, then continue".
How does a system force order only where it is needed?
One obvious way is to wait for the response to every data write before sending the flag. That works if the responses are final ones (lesson 7.4), but it means the producer must track every outstanding write itself. Another obvious way is to put everything on one ordered path, which gives up parallelism for all traffic, not just at the hand-off. A fence gives order exactly at the point it is placed, and nowhere else.
A fence (or barrier) divides a requester's accesses into before and after. No access after the fence may become visible until every access before it has completed to the point the fence promises (accepted on the path, or reached its destination). Between fences, accesses may be reordered freely.
Kind | Orders | Typical use |
|---|---|---|
Write fence (store barrier) | Earlier writes before later writes | Data before flag; descriptors before doorbell |
Full fence | All earlier accesses before all later ones | Hand-offs that mix reads and writes |
Acquire / release | Release: earlier accesses before the releasing write. Acquire: the acquiring read before later accesses | Locks and message passing, at lower cost than full fences |
This is DCLink step 6. The producer posts three data writes into a write buffer that drains one write every 3 cycles, then issues a FENCE. The fence completes only when the buffer is empty; only then does the producer post the flag. The checker confirms the flag reaches memory after all three data writes.
// DCLink step 6: a fence drains a posted-write buffer. The producer posts writes to
// addresses 1, 2 and 3 into a 4-entry write buffer, which drains one write to memory
// every 3 cycles. It then issues a FENCE: the fence completes (fence_cpl) only when
// the buffer is empty, that is, when every earlier write has reached memory. Only
// then does the producer post the flag write (address 9). The checker confirms the
// flag reaches memory after all three data writes.
module tb;
logic clk = 0;
always #5 clk = ~clk;
initial begin $dumpfile("wave.vcd"); $dumpvars(0, tb); end
int cyc = 0;
always @(posedge clk) cyc <= cyc + 1;
logic fence_wait = 0, flag_sent = 0;
// Write buffer: FIFO, drains one entry every 3 cycles; a fence completes when it is empty
logic [3:0] buf_q [0:3];
logic [1:0] rp = 0, wp = 0;
logic [2:0] buf_cnt = 0;
logic [1:0] tick = 0;
wire drain = (buf_cnt != 0) && (tick == 2);
logic mem_wr = 0;
logic [3:0] mem_addr = 0;
wire fence_cpl = fence_wait && (buf_cnt == 0);
// Producer
logic wr;
logic [3:0] wr_addr;
logic fence;
always_comb begin
wr = 0; wr_addr = 0; fence = 0;
case (cyc)
1: begin wr = 1; wr_addr = 1; end
2: begin wr = 1; wr_addr = 2; end
3: begin wr = 1; wr_addr = 3; end
4: fence = 1;
default: ;
endcase
if (fence_cpl && !flag_sent) begin wr = 1; wr_addr = 9; end // flag only after the fence completes
end
always @(posedge clk) begin
tick <= (tick == 2) ? 0 : tick + 1;
if (wr) begin buf_q[wp] <= wr_addr; wp <= wp + 1; end
mem_wr <= drain;
if (drain) begin mem_addr <= buf_q[rp]; rp <= rp + 1; end
buf_cnt <= buf_cnt + wr - drain;
if (fence) fence_wait <= 1;
if (fence_cpl) begin fence_wait <= 0; flag_sent <= 1; end
end
// Checker: count data writes in memory; the flag must find all three there
int landed = 0, errors = 0, flag_at = -1;
always @(posedge clk) if (mem_wr) begin
if (mem_addr == 9) begin if (landed != 3) errors++; flag_at = cyc; end
else landed++;
end
initial begin
#300;
if (errors == 0 && flag_at > 0) $display("PASS: the flag reached memory after all 3 data writes; the fence waited for the buffer to drain");
else $display("FAIL: errors %0d flag_at %0d", errors, flag_at);
$finish;
end
endmoduleThe fence itself is one line: fence_cpl is high when a fence is waiting and the buffer is empty. Everything else is the buffer.
How to read it, slot by slot:
Slots 0 to 2: the producer posts writes to addresses 1, 2 and 3 (wr, wr_addr). The buffer starts draining at once: mem_wr shows address 1 reaching memory in slot 2.
Slot 3: the producer issues the FENCE (fence). Two writes are still in the buffer (buf_cnt 2).
Slots 5 and 8: addresses 2 and 3 reach memory (mem_wr, mem_addr); in slot 8 the buffer is empty and fence_cpl rises.
Slot 8: in the same cycle, and not before, the producer posts the flag (address 9). It drains and reaches memory in slot 11, after all three data writes.
A stall: everything after the fence waits for the slowest access before it, often a full round trip to memory.
Less overlap: accesses that could have proceeded in parallel across the fence cannot.
Implementation: the requester (or its buffers) must know when earlier accesses have completed, which needs final acknowledgements or a drain signal from each buffer on the path.
That is why software uses the weakest fence that is correct (a write fence rather than a full fence, release rather than full ordering) and places it only at hand-offs.
System | Fences |
|---|---|
Processor instruction sets | Arm has DMB and DSB barriers; x86 has MFENCE, SFENCE and LFENCE; RISC-V has FENCE with predecessor and successor sets. Release and acquire forms order only what a lock or flag needs. |
AXI | No fence transaction: a requester waits for earlier write responses (from the destination, if they must have landed) before issuing ordered accesses. |
DCLink (step 6) | A FENCE request whose completion means every earlier write has been performed. |
A driver (illustrative) put a fence between writing descriptors and ringing a doorbell, but used a fence that only ordered accesses to normal memory, while the doorbell was a device register. The processor treated device accesses as outside that fence's scope, and the doorbell occasionally overtook the descriptors. Fences have scopes too (lesson 7.2): the fence must cover both kinds of access it is meant to order.
The fence has not completed: two data writes are still in the buffer (buf_cnt is not 0), so fence_cpl is low.
They order only what a lock needs (accesses inside the critical section against the lock's acquiring read and releasing write), so they stall less than a full fence.
A fence orders everything before it against everything after it, only at the point it is placed.
It works by holding later accesses until earlier ones have completed; every buffer on the path must drain or keep order.
Fences cost a stall; use the weakest correct one, and check its scope covers what it must order.