Common prose words are often single pieces in many splitters. Their groups occurred enough in training to earn long entries.
That is approximate and tokenizer-dependent. Rates vary by model, so measure them on the actual splitter.
Code mixes words with symbols, case changes and underscores. Each boundary is a likely cut, so identifiers split into several pieces.
A Verilog name such as wdata_q_3 has three cut points before the digit. Dots, brackets and widths add further pieces around it.
Shape | Why it splits | Approximate effect |
|---|---|---|
`snake_case` names | Underscores mark boundaries | About one piece per segment, approximately. |
`camelCase` names | Case changes mark boundaries | About one piece per hump, approximately. |
Dotted and indexed access | Each symbol becomes its own piece | One or more extra pieces per symbol. |
Operators and whitespace runs | Dense symbols rarely form long groups | Short pieces throughout. |
Digit strings split into short pieces, and hex such as 8'hff splits at the quote and case steps. Long numbers therefore cost more pieces than a prose word of equal length.
Non-Latin scripts average fewer characters per token with common splitters, approximately. This too varies by model, so measure it on representative samples.
44 characters of prose at about 4 chars per token (stated approximate rate)
tokens = 44 / 4 = 11
44 characters of dense code at about 2.5 chars per token (stated approximate rate)
tokens = 44 / 2.5 = 17.6, about 17 to 18 pieces
ratio = 17.6 / 11 = 1.6, so the code uses about three-fifths more piecesThe program below uses a FAKE scripted splitter, not any real tokenizer. It cuts on every non-alphanumeric character to show boundaries, and says so.
import sys
PROSE = "the cat sat on the mat"
CODE = "wdata_3<=8'hff;"
def fake_split(s):
parts = []
cur = ""
for ch in s:
if ch.isalnum():
cur += ch
else:
if cur:
parts.append(cur)
cur = ""
if ch.strip():
parts.append(ch)
if cur:
parts.append(cur)
return parts
bad = 0
for s in [PROSE, CODE, "clk"]:
parts = fake_split(s)
if any(p == "" for p in parts):
bad += 1
if "".join(parts) != s.replace(" ", ""):
bad += 1
if len(fake_split(CODE)) <= len(fake_split(PROSE)):
bad += 1
if fake_split("clk") != ["clk"]:
bad += 1
if bad != 0:
print("FAIL: " + str(bad) + " checks wrong")
sys.exit(1)
print("PASS: prose " + str(len(fake_split(PROSE))) + " parts code " + str(len(fake_split(CODE))) + " parts")wdata_q_3 is 9 characters. Why is a 2-piece guess optimistic?Two underscores plus a digit boundary invite cuts, so about one piece per segment is likelier. Exact splits still vary by model.
It proves boundaries multiply pieces on code-like shapes. It proves nothing about any real splitter, whose table and rates vary by model.