Separate core IPC from system traffic

Core IPC looks healthy while NoC bandwidth and system power explode. Per-core metrics hide a system traffic problem.

Assume 64 cores sharing a counter array, one 64-byte line, and a storm mix that sums to 100 percent. Coherence is the largest class.

1. 0-5 min: split core IPC from fabric and power counters.

2. 5-15 min: measure coherence transaction rate and invalidation clusters.

3. 15-25 min: map hot lines to source layout.

4. 25-35 min: compare software layout vs hardware policy fixes.

5. 35-45 min: validate on application traces, not microbenchmarks.

Prove false sharing with line math

Eight 8-byte counters pack into one 64-byte line. Each core writes only its counter, yet every write moves the whole line.

At 10M increments per second per core over 64 cores, line traffic is 640M x 64 B = 41 GB/s. Useful bytes are 640M x 8 B = 5.1 GB/s, an 8 to 1 overhead.

Increments: 64 cores x 10M per s = 640M line moves per s
Traffic:   640M x 64 B = 41 GB/s of coherence traffic
Useful:    640M x 8 B = 5.1 GB/s of counter bytes
Overhead:  64 / 8 = 8 to 1, all false sharing

Pad and align the hot layout

Pad each counter to 64 bytes and align the array to a line. Each core then owns its line in Modified state and the bouncing stops.

Line traffic falls from 41 GB/s toward zero plus true sharing only. One layout edit beats any fabric upgrade.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

module line_next (
  input  logic [1:0] cur,
  input  logic       rd,
  input  logic       wr,
  input  logic       other_wr,
  output logic [1:0] nxt
);
  localparam [1:0] ST_I = 2'b00, ST_E = 2'b01, ST_M = 2'b10;
  always_comb begin
    nxt = cur;
    if (cur == ST_I) begin
      if (wr) nxt = ST_M;
      else if (rd) nxt = ST_E;
    end else if (cur == ST_E) begin
      if (wr) nxt = ST_M;
      else if (other_wr) nxt = ST_I;
    end else if (cur == ST_M) begin
      if (other_wr) nxt = ST_I;
    end
  end
endmodule

The testbench walks the exclusive-owned subset used in this storm. Reads take Invalid to Exclusive, writes take Exclusive to Modified, and another writer returns either state to Invalid.

Match the protocol to the sharing

Snoop fits small uniform domains with cheap broadcast. A directory fits 64 sharers with targeted invalidations instead of broadcast.

CHI is a separate protocol with decoupled transaction channels, not a revision of ACE. Name the one the SoC uses and its invalidation path.

Related topics

• Cache coherence for the architect

• On-chip networks