All rates below are approximate and tokenizer-dependent. They describe common splitters, and exact values vary by model, so measure them.
Material | Approximate chars per token | Reading |
|---|---|---|
Plain English prose | About 3 to 4 | Long words often survive whole. |
Code and Verilog | About 2 to 3 | Symbols and names split shorter. |
Hex and dense numbers | About 2 or fewer | Digits rarely form long groups. |
Non-Latin text | Fewer than prose | More pieces per character, approximately. |
The calc block converts one 12000-character file at prose and code rates. Ranges replace single numbers throughout.
file size = 12000 characters
prose at 4 chars per token: 12000 / 4 = 3000 tokens
prose at 3 chars per token: 12000 / 3 = 4000 tokens
prose range: 3000 to 4000 tokens (approximate)
code at 3 chars per token: 12000 / 3 = 4000 tokens
code at 2 chars per token: 12000 / 2 = 6000 tokens
code range: 4000 to 6000 tokens (approximate)Mix sets the rate: comments read like prose while identifiers read like code. Whitespace, casing and symbol density move any file along the range.
Material mix: comments raise the average, identifiers and literals lower it.
Formatting: dense one-line code splits shorter than spaced, commented code.
Splitter choice: tables differ, so the same file rates differently across models.
Exactly is wrong. The rate is an approximate range that varies by model and file, so 2.5 can only be a stated rate for one estimate.
Neither endpoint alone. Estimate each half at its own rate and add, or measure a representative sample directly.