Local to which part of a miss?

NUMA means non-uniform memory access: a miss costs more when it has to cross to another node. In the course model and in classic CC-NUMA, the home of X sits at the memory node that holds X's physical page: the directory sits beside that node's memory controller, so placing the page places the home. Use two nodes, A and B, each with its own memory and home. C0 and C1 are both on node A. The dirty-responsible cache is the third location that matters: when it holds X in M, H must fetch the value from it, wherever the page lives.

A design may separate the two, for example a directory slice chosen by an address hash while memory channels are interleaved differently. That is an extension: a clean miss then pays the requester-to-home legs and a home-to-memory access as separate costs. This lesson keeps home and memory together.

Worked latency comparison

Take Anatomy of a Miss Latency's 3 cycles per in-node leg, 2 for H's lookup and 1 for a cache to handle a forward. Drop its 4-cycle request queue and 1-cycle grant construction, which are the same in every case here. Use 12 for a leg that crosses between nodes and 10 instead of Anatomy's 30 for H's memory read, so that a memory read costs about one remote leg. Use MSI. C0 issues a GetS for X, which started at memory = 0. In the dirty cases C1 has stored X = 1 and holds M / 1, H exclusive(C1), and the read takes the four-hop route of Three-Hop Versus Four-Hop. H's memory update for the dirty read is left off C0's path.

The remote-dirty case is the CC-NUMA trap: the newest value sits in C1, on C0's own node, yet every leg crosses to the home on node B and back. Moving the page to node A turns 51 cycles into 15, cheaper even than the local clean read, because no memory read is needed. The ordering depends on the leg costs: with uniform 3-cycle legs, both page placements cost the same.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# Illustrative cycles, not measurements. Legs as in Anatomy of a Miss Latency,
# simplified: its request queue and grant construction are dropped (equal in
# every case) and its 30-cycle memory read is cut to 10.
LOOKUP, HANDLE, MEMORY = 2, 1, 10
def clean(leg):                     # GetS, H lookup, memory read, Data
    return leg + LOOKUP + MEMORY + leg
def dirty(to_home, home_to_holder):  # four-hop: GetS, Fwd-GetS, Data to H, Data
    return to_home + LOOKUP + home_to_holder + HANDLE + home_to_holder + to_home
def cases(local, remote):           # C0 and C1 on node A, requester C0
    return {
        "local-home clean": clean(local),            # X's page on node A
        "remote-home clean": clean(remote),          # X's page on node B
        "remote-home dirty": dirty(remote, remote),  # page on B, C1 holds M
        "migrated dirty": dirty(local, local),       # page moved to node A
    }
c = cases(local=3, remote=12)
assert c == {"local-home clean": 18, "remote-home clean": 36,
             "remote-home dirty": 51, "migrated dirty": 15}
assert c["local-home clean"] < c["remote-home clean"] < c["remote-home dirty"]
assert c["migrated dirty"] < c["local-home clean"]  # neighbour beats memory
u = cases(local=3, remote=3)        # uniform legs: placement stops mattering
assert u["local-home clean"] == u["remote-home clean"]
assert u["remote-home dirty"] == u["migrated dirty"]
print("PASS: remote home 36 and remote dirty 51 cycles versus 18 local, 15 after migration")

First-touch placement and page migration

Most operating systems place a page on first touch: the page is allocated from the memory of the node whose core first faults it (Linux's default policy allocates on the faulting CPU's local node). The classic mistake is a single initialisation thread on node B that touches a whole array before the worker threads on node A start: every page, and so every home, lands on node B, and every miss the workers take is a remote case from the table. The fix is to initialise the data with the same threads, on the same nodes, that will later use it, or to bind the memory explicitly.

Page migration repairs a bad placement later. The OS (for example Linux's automatic NUMA balancing, or an explicit move_pages call) copies the page to a frame on the node that uses it, updates the page table and invalidates the stale TLB entries. The copy reads every line coherently, so a line held dirty in a cache is supplied by its holder, not by stale memory. The new frame has a new physical address and therefore a new home; coherence moves nothing, the old home simply stops receiving requests for that data. A 4 KiB page is 64 lines plus a TLB shootdown, so migration pays off only for a page that will take many more misses at its new node.

A placement experiment should keep the logical work constant while changing one policy at a time: page placement, thread affinity or data layout. Record memory traffic separately from cache-to-cache traffic. For a single truly shared X written by cores on both nodes, no placement removes the remote legs for every writer.

Related topics

• Shared course model

• Leg-by-leg latency

• Data authority before proximity