What happens to GetM when C0 and H have different clocks?

A message is many bits: type, address, transaction identity and sometimes a 64-byte payload. Sampling those bits directly in another clock domain can mix old and new values. The standard structure is an asynchronous FIFO: the sender writes the whole message into a storage slot in its own clock domain, and only the FIFO pointers cross. Each pointer is Gray-coded, so one bit changes per increment, and passes through a two-flop synchronizer in the other domain. The receiver reads a slot only after the synchronized write pointer shows it is filled; by then the slot's bits have been stable for at least two receiver cycles. The async FIFO lessons and Gray pointers in sign-off teach the design; here it is a box with one rule.

The rule for coherence: one asynchronous FIFO per message class and direction. Cluster to H carries requests (GetS, GetM, PutS, PutE, PutM) and responses (Inv-Ack, Data from a forwarding cache). H to cluster carries forwards (Inv, Fwd-GetS, Fwd-GetM) and responses (Data, Put-Ack). That is four FIFOs. Within a class the FIFO keeps order, but the protocol must not depend on it (the course assumes no FIFO order), and classes may overtake each other freely.

The same care applies to the coherency-entry handshake that sits beside the channels. The CHI SYSCOREQ/SYSCOACK handshake signals must be synchronous to the interface clock, and a bridge that crosses a clock domain must synchronize them (Arm IHI 0050 Issue E.a, section 15.1).

Why a shared FIFO brings the deadlock back

Part 5's rule is that responses always sink: consuming a response never waits on a resource that a request can hold. A Protocol Deadlock broke its P/Q cycle by giving X's Data a reserved slot Qresp. Now put that design behind a crossing. If the cluster-to-H direction uses one shared FIFO, the FIFO itself is the old shared Q: it is strictly in order, so H can read only its head.

Keep the roles of the linked lesson (MSI, X warm-up: C0 stored 1, memory = 0). C1's GetM(X) holds H's only pending entry P, C0 answered Fwd-GetM and gave up X, and C0's next miss GetS(Y) entered the crossing before C0's Data(1). The cache states at that checkpoint are exactly the linked lesson's step 3:

The same holds in the other direction: a forward stuck behind a blocked response, or a response behind a forward the cache cannot accept yet, rebuilds a cycle through the cache controller. Separate FIFOs are necessary but, as in Message Classes and Virtual Networks, not sufficient: the response FIFO also needs space when a response arrives. That is a credit question, the next lesson.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

def run(order, shared, depth):
    """Cluster-to-H crossing. P is held by C1's GetM(X) until Data X arrives."""
    fifos = {"req": [], "resp": []}
    waiting = list(order)               # messages the cluster still has to write
    p_free, done = False, set()
    def fifo_of(m):
        return "req" if shared or m == "GetS Y" else "resp"
    while True:
        moved = False
        while waiting and len(fifos[fifo_of(waiting[0])]) < depth:
            m = waiting.pop(0)
            fifos[fifo_of(m)].append(m)
            moved = True
        for name in ("resp", "req"):    # H looks only at each FIFO's head
            if not fifos[name]:
                continue
            head = fifos[name][0]
            if head == "Data X":        # response: uses the existing P
                fifos[name].pop(0)
                p_free = True
                done.add("X")
                moved = True
            elif head == "GetS Y" and p_free:
                fifos[name].pop(0)
                done.add("Y")
                moved = True
        if not moved:
            return done
orders = [("GetS Y", "Data X"), ("Data X", "GetS Y")]
for depth in range(1, 9):
    for order in orders:
        assert run(order, shared=False, depth=depth) == {"X", "Y"}
    # failing case: one shared FIFO, request ahead of the response
    assert run(orders[0], shared=True, depth=depth) == set()
    assert run(orders[1], shared=True, depth=depth) == {"X", "Y"}
print("PASS: per-class FIFOs complete X and Y, a shared FIFO of any depth can deadlock")

The check runs depths 1 to 8. Separate FIFOs complete both transactions in both arrival orders. The shared FIFO completes only when the response happens to be ahead, and never when GetS(Y) is at the head, at any depth.

Latency: what the synchronizers add to a miss

A message written into an asynchronous FIFO becomes readable only after the write pointer has passed the receiver's two synchronizer flops, plus up to one more receiver cycle because the clocks have no fixed phase. The full and empty flags are therefore conservative: the receiver sees a new entry late, the sender sees a freed slot late, never early. Take an illustrative 3 cycles per crossing, in the same illustrative cycle unit as Anatomy of a Miss Latency.

Every serial leg pays the crossing once, so a protocol with more serial legs pays more. A three-hop transfer between C0 and C1 inside the cluster domain pays for the request and forward legs only, since the Data leg stays in the cluster; the H-collected upgrade pays four times. Where the clocks have a known fixed ratio, designers can use cheaper crossings, but that is a clocking decision, not a protocol one.

Related topics

• Shared course model

• The P/Q cycle this lesson moves into the crossing

• CDC course: asynchronous FIFOs

• CDC course: two-flop synchronizers

• Latency breakdown being extended