Raw log lines differ on every run: timestamps, ids, addresses and counts change while the failure stays the same. Counting raw lines therefore reports every line as unique.
Normalisation replaces selected varying fields with placeholders so similar messages can be counted. A message group is not necessarily one root cause. Preserve raw lines and identifiers: addresses, lanes, and durations may distinguish failures that a broad normalisation merges.
The program below clusters a synthetic log by message pattern. Its broad number replacement demonstrates grouping mechanics, not a production parser. If run, its self-checks would report the expected counts; no execution is claimed here.
import re
import sys
LOGS = [
"12:01:03 TIMEOUT req id 4821 addr 0x7f3a after 500 ms",
"12:01:04 TIMEOUT req id 4822 addr 0x7f40 after 500 ms",
"12:01:05 TIMEOUT req id 4823 addr 0x7f46 after 512 ms",
"12:01:09 TIMEOUT req id 4829 addr 0x8001 after 500 ms",
"12:01:11 TIMEOUT req id 4835 addr 0x8012 after 500 ms",
"12:01:14 TIMEOUT req id 4840 addr 0x8023 after 501 ms",
"12:01:19 TIMEOUT req id 4851 addr 0x8034 after 500 ms",
"12:02:01 PARITY lane 0 pkt 99182",
"12:02:02 PARITY lane 2 pkt 99183",
"12:02:07 PARITY lane 0 pkt 99210",
"12:03:10 LINK down port 3",
"12:03:11 LINK down port 5",
]
def normalize(line):
line = re.sub(r"0x[0-9a-fA-F]+", "0xADDR", line)
line = re.sub(r"[0-9]+", "N", line)
return line
def main():
groups = {}
for line in LOGS:
key = normalize(line)
groups[key] = groups.get(key, 0) + 1
if len(LOGS) != 12:
print("FAIL: line total is %d" % len(LOGS))
sys.exit(1)
if len(groups) != 3:
print("FAIL: group total is %d" % len(groups))
sys.exit(1)
counts = sorted(groups.values())
if counts != [2, 3, 7]:
print("FAIL: counts are %s" % (counts,))
sys.exit(1)
top = max(groups, key=groups.get)
if "TIMEOUT" not in top:
print("FAIL: top group is %s" % top)
sys.exit(1)
print("PASS: 12 lines form 3 groups with counts 7, 3 and 2")
main()Twelve lines become three groups with counts 7, 3 and 2. The table below reads each group, following the program above line for line.
Group | Lines | What to check next |
|---|---|---|
TIMEOUT normalised message | 7 | Slice the first TIMEOUT and rank hypotheses for the req path. |
PARITY normalised message | 3 | Check whether the lane events overlap the TIMEOUT window. |
LINK down normalised message | 2 | Confirm the two ports, then decide if this is separate or fallout. |
cluster counts: 7, 3 and 2
total = 7 + 3 + 2 = 12
shares, rounded to two decimals: 7 / 12 = 58.33%, 3 / 12 = 25.00%, 2 / 12 = 16.67%
check: 58.33 + 25.00 + 16.67 = 100.00
the top group holds over half the lines; frequency alone does not establish
severity, root cause, or investigation priorityExercise: inspect the original PARITY lines after grouping. Lane 0 appears twice and lane 2 once, but the normalized key removes that distinction. Keep lane as a structured field if the investigation asks whether one lane dominates. Likewise, retain request IDs for cross-log matching and durations for timeout-boundary analysis.
Normalisation. Ids, addresses and timestamps differ on every line, so raw lines never repeat. Replacing them with placeholders reveals the 3 repeated messages.
No. It is the largest message group, but the original LINK-down events or a severe parity event may explain other symptoms. Inspect chronology, severity, and dependencies before deciding; counts do not identify a cause.