A 4-core system keeps its caches coherent by broadcasting every cache miss to every other core: "does anyone have this line?". It works well. The same design scaled to 64 cores sends 63 snoops and collects 63 answers for every miss, and the snoop traffic alone exceeds the bandwidth the fabric was built for. Nothing was wrong at 4 cores. Broadcast grows with the system, and the data it carries does not.
What does it cost to send one message to many agents, and how do large systems avoid it?
When a message concerns everyone, or you do not know who it concerns, the obvious answer is to send it to everyone. On a shared bus that is free: every agent sees every transaction anyway. On a point-to-point fabric each copy is a separate message on separate links, often with a separate response to collect, so the cost per event grows with the number of agents, and so does the rate of events. Total traffic grows roughly with the square of the system size.
Method | How | Cost per event | Scales? |
|---|---|---|---|
Broadcast | Send to every agent | N − 1 messages (and often N − 1 responses) | Poorly: grows with N |
Multicast | Send to a chosen set, replicating at branch points in the network | Only the targets, shared links carry one copy | Better, but needs the set and replication support |
Filtering (directory) | Keep track of who actually needs the message, and send only to them | Usually 0 to a few messages | Well; costs the tracking state |
Illustrative: every core misses its cache 20 million times per second, and each snoop and answer is one 16-byte message.
Cores: 4 64
Snoops per miss (broadcast): 3 63
Snoop messages per second: 4 x 20e6 x 3 = 0.24e9 64 x 20e6 x 63 = 80.6e9
Plus answers (same count), 16 bytes each: ~7.7 GB/s ~2.6 TB/s
With a directory (say 0.2 holders per miss on average): 64 x 20e6 x 0.2 x 2 x 16 B = ~8.2 GB/sCoherence snoops and invalidations (Part 13): "who has this line?" and "drop your copy". Snooping protocols broadcast; directory protocols filter.
Interrupts to several cores and inter-processor interrupts: usually multicast by an interrupt controller with a target mask.
Barriers and synchronisation: "everyone has arrived" collected from many agents, often with a tree to combine answers.
Configuration and power broadcasts: rare enough that broadcast is fine.
Collecting answers is often the harder half: a broadcast that needs a response from every target must wait for the slowest, and the requester must track N − 1 outstanding replies (Part 6).
System | Approach |
|---|---|
Small coherent systems on a shared bus | Broadcast snooping: every cache watches every transaction. |
Large coherent fabrics | Directories or snoop filters that track which caches may hold each line, so most misses need no snoop at all, and the rest go only to the holders. |
On-chip networks | Multicast support in routers (replicating at branch points) for the few message types that need many targets. |
The opening system (illustrative): broadcast snooping scaled from 4 to 16 cores on a mesh, and snoop traffic rose about 20 times while useful data traffic rose 4 times. Latency to memory rose with it, because snoops competed for the same links. A snoop filter that tracked which cores held each line removed most snoops, and the latency fell back.
62: 31 requests out and 31 answers back.
On a bus every agent already sees every transaction; on a mesh each copy is a separate message over separate links.
Broadcast costs N − 1 messages (and often N − 1 answers) per event, and events grow with N too.
Multicast sends only to a chosen set; directories and snoop filters send only where needed.
Keep true broadcast for rare events; filter or multicast anything frequent at scale.