Read the PMU before touching the RTL

The symptom is L2 MPKI doubling from 8 to 16 after an OS page policy change. Assume CPI_core 1.5 excluding L2-miss stalls and a 40-cycle L2 miss penalty through a mostly-hit L3.

The storm stall breakdown below sums to 100 percent. L2-miss stalls dominate, so the debug starts at the L2 and its page mapping.

1. 0-5 min: restate workload, baseline MPKI 8 and the page policy change.

2. 5-15 min: split MPKI by level and add TLB MPKI plus bandwidth.

3. 15-25 min: rank footprint growth, mapping conflicts, replacement and prefetch.

4. 25-35 min: propose one experiment per hypothesis, cheapest first.

5. 35-45 min: fix, then validate on traces with a PPA note.

Separate capacity from conflict

A doubled MPKI after a software mapping change points at conflicts, not capacity. Capacity responds to footprint, conflicts respond to placement.

The model below is a 4-entry direct-mapped cache over the 8-access trace 00, 04, 00, 04, 01, 01, 02, 02. Addresses 00 and 04 share index 0 with different tags, so they evict each other every time.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

module dm4 (
  input  logic       clk,
  input  logic       rst_n,
  input  logic       req,
  input  logic [7:0] addr,
  output logic       hit
);
  logic [5:0] ctag [4];
  logic       cvalid [4];
  logic [1:0] idx;
  logic [5:0] tag;
  assign idx = addr[1:0];
  assign tag = addr[7:2];
  always @(posedge clk or negedge rst_n) begin
    if (!rst_n) begin
      cvalid[0] <= 1'b0; cvalid[1] <= 1'b0;
      cvalid[2] <= 1'b0; cvalid[3] <= 1'b0;
      ctag[0] <= 6'd0; ctag[1] <= 6'd0;
      ctag[2] <= 6'd0; ctag[3] <= 6'd0;
      hit <= 1'b0;
    end else begin
      hit <= req && cvalid[idx] && (ctag[idx] == tag);
      if (req) begin
        ctag[idx] <= tag;
        cvalid[idx] <= 1'b1;
      end
    end
  end
endmodule

The testbench proves the conflict story. Two addresses on one index produce zero hits across four alternating accesses, while untouched indexes hit on repeat.

The worked sizing

Base stalls are 8 x 40 / 1000 = 0.32 CPI, so CPI is 1.82 and IPC is 0.55. Storm stalls are 16 x 40 / 1000 = 0.64 CPI, so CPI is 2.14 and IPC is 0.47.

The IPC drop is 1 - 0.47 / 0.55 = 0.15, about 15 percent. Miss rate in the model is 6 of 8 = 75 percent, so its AMAT is 1 + 0.75 x 10 = 8.5 cycles.

Base:  CPI = 1.5 + 8 x 40 / 1000 = 1.82,  IPC = 1 / 1.82 = 0.55
Storm: CPI = 1.5 + 16 x 40 / 1000 = 2.14, IPC = 1 / 2.14 = 0.47
Drop:  1 - 0.47 / 0.55 = 0.15, about a 15 percent IPC drop
Model: 6 misses of 8, AMAT = 1 + 0.75 x 10 = 8.5 cycles

Fix the mapping, not the size

Doubling the L2 would soften conflicts without removing them. Restoring the page interleave or coloring attacks the cause directly.

If the fix removes 7 of the 8 added MPKI, stalls return to 9 x 40 / 1000 = 0.36 CPI. The validation lesson checks that endpoint.

Related topics

• Caches

• Memory bandwidth and latency

• Performance method